echogarden 0.5.0 → 0.5.2

package/README.md

@@ -1,6 +1,6 @@
 # Echogarden
 
-A fully open-source speech system designed with end-users in mind.
+Echogarden is an integrated speech system that provides a range of speech generation, recognition and processing tools that are designed to be directly usable by end-users:
 
 * Written in TypeScript, for the Node.js runtime
 * Easy to install, run and update
@@ -12,32 +12,15 @@ A fully open-source speech system designed with end-users in mind.
 
 * Fast, high-quality offline text-to-speech voices based on the [VITS](https://github.com/jaywalnut310/vits) neural architecture
 * Accurate offline speech recognition using [OpenAI Whisper](https://openai.com/research/whisper) models
-* Supports synthesis and recognition via major cloud providers, including Google, Microsoft and Amazon
+* Supports synthesis and recognition via a [variety of offline and cloud engines](docs/Engines.md), including Google, Microsoft and Amazon
 * Word-level timestamps for all synthesis and recognition outputs
 * Speech-to-transcript alignment using dynamic time warping (DTW), and dynamic time warping with recognition assist (DTW-RA) methods
 * Advanced subtitle generation, accounting for sentence and phrase boundaries
-* Can transcribe speech in any one of 98 languages, translated directly to English, and produce near word-level synchronized subtitles for the translated transcript
+* Can translate speech in any one of 98 languages, transcribe it directly to English, and produce near word-level synchronized subtitles for the translated transcript
 * Attempts to improve TTS pronunciation accuracy for a few engines and languages (currently only implemented for English dialects): adds text normalization (e.g. idiomatic date and currency pronunciation), heteronym disambiguation (based on a custom rule-based model) and user-customizable pronunciation lexicons
 * Internal package system that auto-downloads and installs voices, models and other resources, as needed
 * Other features include: language detection (both for audio and text), voice activity detection, and speech denoising
 
-### In development
-
-* Background worker
-* WebSocket-based server and API
-* Browser extension (for TTS only), including integration with the Web Speech API, and an advanced page reader enabling real-time narration of any page content, with live word highlighting
-* New, high-accuracy text language identification model (own work)
-
-### Planned, but not yet
-
-* Text enhancement, adding breaks to improve phrasing of synthesized text, as well as adding missing punctuation to recognized transcripts, if needed
-* Web-based UI
-* Real-time, streaming speech recognition
-
-### Maybe
-
-* Browser port for a subset of the API (in particular for the offline TTS models and their dependencies)
-
 ## Installation
 
 Ensure you have [Node.js](https://nodejs.org/) `v16.0.0` or later installed.
@@ -48,7 +31,7 @@ npm install echogarden -g
 ```
 
 Additional tools:
-* [`sox`](https://sourceforge.net/projects/sox/): used for the CLI's audio playback and recording (only). Auto-installed via a package on Windows and Intel macOS. On Linux and ARM64 macOS, it is recommended to install it via platform package managers like `apt` and `brew`.
+* [`sox`](https://sourceforge.net/projects/sox/): used for the CLI's audio playback and recording (only). Auto-installed via an expansion package on Windows and Intel macOS. On Linux and ARM64 macOS, it is recommended to install it via platform package managers like `apt` and `brew`.
 * [`ffmpeg`](https://ffmpeg.org/download.html): used for codec conversions. Auto-installed via an expansion package on Windows, Intel macOS, and x64 Linux. On ARM64 macOS, it is recommended to install it via platform package manager like `brew`, otherwise, much slower `ffmpeg-wasm` would be used.
 
 (hopefully in the future all platforms would be covered using expansion packages)
@@ -59,15 +42,43 @@ Additional tools:
 npm update echogarden -g
 ```
 
-## Next steps
+## Interfacing with the system
+
+Currently, the software is operated mainly through a [command-line interface](docs/CLI.md), which enables powerful customization and is especially useful for long-running bulk operations.
+
+Development of more graphical and interactive tooling is currently ongoing. A general roadmap is shown further down below.
+
+## Guides and resource pages
 
-* [Using the command line interface](docs/CLI.md)
+* [Using the command-line interface](docs/CLI.md)
 * [Options reference](docs/Options.md)
 * [Full list of supported engines](docs/Engines.md)
 * [Technical overview and Q&A](docs/Technical.md)
-* [Roadmap](docs/Roadmap.md)
+* [Developer's task list](docs/Tasklist.md)
 * [How to help](docs/Development.md)
 
+## Development roadmap
+
+_(For much more detailed information, see the [developer task list](docs/Tasklist.md))_.
+
+### In development
+
+* (**90**%) Background worker
+* (**90**%) WebSocket-based server and API
+* (**70**%) Browser extension, acting as a bridge to a local or self-hosted server. Includes integration with the Web Speech API, and real-time narration of page content, with live word highlighting
+* (**60**%) New, high-accuracy text language identification model (own work)
+
+### Planned, but not yet started
+
+* Web-based UI frontend
+* Text enhancement, adding breaks to improve phrasing of synthesized text, as well as adding missing punctuation to recognized transcripts, if needed
+* Real-time, streaming speech recognition
+* Official developer API (it's currently possible to import the `echogarden` npm package as a library, but the official API has not been published yet)
+
+### Future (maybe)
+
+* Browser port for a subset of the API (in particular for the offline TTS models and their dependencies)
+
 ## Credits
 
 This project consolidates, and builds upon the effort of many different individuals and companies, as well as contributing a number of original works.

package/data/lexicons/heteronyms.en.json

@@ -104,7 +104,7 @@
 			],
 
 			"succeededBy": [
-				"itself", "us", "opportunities", "them", "his", "her", "our", "my", "its", "this", "no", "some", "another", "different", "various", "several", "many", "few", "other", "unique", "special", "great", "excellent", "nice"
+				"itself", "us", "opportunities", "them", "a", "an", "his", "her", "our", "my", "its", "this", "no", "some", "another", "different", "various", "several", "many", "few", "other", "unique", "special", "great", "excellent", "nice"
 			],
 
 			"notSucceededBy": [

package/docs/CLI.md

@@ -10,6 +10,10 @@ Here's a quick tour of the main operations available via the CLI.
 
 Each command can accepts one or more options, in the form `--[optionName]=[value]` (The `=` is required). A detailed reference of all the available options can be found [here](Options.md).
 
+**Keyboard shortcuts**:
+* While the program is running, you can press `esc` to exit immediately
+* When audio is playing, you can press `enter` to skip it
+
 ## Text to speech
 
 **Task**: given a text file, synthesize spoken audio for it.
@@ -44,12 +48,12 @@ The CLI supports multiple output files. This would synthesize a text file, and s
 echogarden speak-file text.txt result.mp3 result.wav result.srt --engine=vits --speed=1.1
 ```
 
-Synthesize a web page (will try to extract its main article parts and omit the rest):
+Synthesize a web page (it will try to extract its main article parts and omit the rest):
 ```bash
 echogarden speak-url https://example.com/hola
 ```
 
-Synthesize a Wikipedia article in any of its language editions:
+Synthesize a Wikipedia article, in any of its language editions:
 ```bash
 echogarden speak-wikipedia "Psychologie" --language=fr
 ```
@@ -198,7 +202,7 @@ Try to identify the language of a text file, and print the probabilities to the
 echogarden detect-text-language story.txt
 ```
 
-Try to identify the language of a text file, and store the probabilities in a JSON file:
+Try to identify the language of a text file, and store the detailed probabilities in a JSON file:
 ```bash
 echogarden detect-text-language story.txt detection-results.json
 ```

package/docs/Development.md

@@ -4,6 +4,16 @@ So far, this project has been the solo work of a single person (yours truly).
 
 However, there are many areas where contributions can be made.
 
-For instance, if you are proficient in one or more languages, other than English, you could help a lot by extending the pronunciations lexicons to help correct errors in the default phonemizations produced by the eSpeak engine, or add rules to help resolve the pronunciations of heteronyms (words that are written the same but can be pronounced in different ways depending on their context) based on their context.
+## Reporting any issue or bug you encounter
 
-... TODO ...
+Especially if you're using the macOS platform, since I don't have access to a macOS machine.
+
+## Reporting odd TTS pronunciations and other fail cases
+
+Though that may be an issue with model training, which should be forwarded to the original authors
+
+## Extending the pronunciation lexicons
+
+Especially if you are proficient in a language other than English. In many cases, the default phonemizations produced by the eSpeak engine are incorrect
+
+You can also add rules to help resolve the pronunciations of heteronyms (words that are written the same but can be pronounced in different ways depending on their context) based on their context.

package/docs/Engines.md

@@ -4,7 +4,7 @@
 ## Text to speech
 
 **Offline**:
-* [VITS](https://github.com/jaywalnut310/vits) (`vits`): a high-quality end-to-end neural speech synthesis architecture. Currently available models were trained by Michael Hansen as part of his [Piper speech synthesis system](https://github.com/rhasspy/piper). Currently there are 83 voices, in a range of languages, including English (US, UK), Spanish (ES, MX), Brazilian Portuguese, Italian, French, German, Dutch (NL, BE), Swedish, Norwegian, Danish, Finnish, Polish, Greek, Russian, Ukrainian, Catalan, Icelandic, Swahili, Kazakh, Georgian, Nepali, Vietnamese and Chinese. You can listen to audio samples of all voices and languages in [Piper's samples page](https://rhasspy.github.io/piper-samples/).
+* [VITS](https://github.com/jaywalnut310/vits) (`vits`): a high-quality end-to-end neural speech synthesis architecture. Available models were trained by Michael Hansen as part of his [Piper speech synthesis system](https://github.com/rhasspy/piper). Currently there are 83 voices, in a range of languages, including English (US, UK), Spanish (ES, MX), Brazilian Portuguese, Italian, French, German, Dutch (NL, BE), Swedish, Norwegian, Danish, Finnish, Polish, Greek, Russian, Ukrainian, Catalan, Icelandic, Swahili, Kazakh, Georgian, Nepali, Vietnamese and Chinese. You can listen to audio samples of all voices and languages in [Piper's samples page](https://rhasspy.github.io/piper-samples/).
 * [SVOX Pico](https://github.com/naggety/picotts) (`pico`): a legacy diphone-based synthesis engine. Supports English (US, UK), Spanish, Italian, French, and German.
 * [Flite](https://github.com/festvox/flite) (`flite`): a legacy diphone-based synthesis engine. Supports English (US, Scottish), and several Indic languages: Hindi, Bengali, Marathi, Telugu, Tamil, Gujarati, Kannada and Punjabi.
 * [eSpeak-NG](https://github.com/espeak-ng/espeak-ng/) (`espeak`): a lightweight "robot" sounding formant-based synthesizer. Supports 100+ languages. Extensively used internally for speech alignment, phonemization, and other internal tasks.
@@ -59,8 +59,8 @@ These are commercial services that require a subscription and an API key to use:
 
 These engines' goal is to match (or "align") a given spoken recording with a given transcript as closely as possible. They will annotate each word in the transcript with approximate start and end timestamps:
 
-* Dynamic Time Warping (`dtw`): transcript is first synthesized using the `espeak` engine, then [DTW](https://en.wikipedia.org/wiki/Dynamic_time_warping) is applied to find the best mapping between the synthesized audio and the original audio.
-* Dynamic Time Warping with Recognition Assist (`dtw-ra`): recognition is applied to the audio (any recognition engine can be used), then both the ground-truth transcript and the recognized transcript are synthesized using `espeak`. Then, the best mapping is found between the two synthesized audio sequences, and the result is mapped back to the original audio using the timing information produced by the recognizer.
+* Dynamic Time Warping (`dtw`): transcript is first synthesized using the eSpeak engine, then [DTW](https://en.wikipedia.org/wiki/Dynamic_time_warping) is applied to find the best mapping between the synthesized audio and the original audio.
+* Dynamic Time Warping with Recognition Assist (`dtw-ra`): recognition is applied to the audio (any recognition engine can be used), then both the ground-truth transcript and the recognized transcript are synthesized using eSpeak. Then, the best mapping is found between the two synthesized audio sequences, and the result is mapped back to the original audio using the timing information produced by the recognizer.
 * Whisper-based alignment (`whisper`): transcript is tokenized and decoded along with the audio using the Whisper model, then timestamps are extracted from the internal state of the model (_note_: currently, only supports audio inputs that are 30 seconds or less).
 
 ## Speech translation
@@ -70,8 +70,8 @@ These engines' goal is to match (or "align") a given spoken recording with a giv
 ## Language detection
 
 **Spoken language detection**:
-* [Whisper](https://github.com/openai/whisper) (`whisper`): uses the language token produced by the `whisper` speech recognition model to generate a set of probabilities for the 99 languages it has been trained on.
 * [Silero Language Classifier](https://github.com/snakers4/silero-vad/wiki/Other-Models) (`silero`): a speech language classification model by Silero.
+* [Whisper](https://github.com/openai/whisper) (`whisper`): uses the language token produced by the `whisper` speech recognition model to generate a set of probabilities for the 99 languages it has been trained on (_note_: currently only uses the first 30 seconds of the audio).
 
 **Text language detection**:
 * [TinyLD](https://www.npmjs.com/package/tinyld) (`tinyld`): a simple language detection library.

package/docs/Options.md

@@ -1,6 +1,10 @@
 # Configuration options reference
 
-For a comprehensive list of all supported engines: see [this page](Engines.md).
+Here is a detailed reference for the options accepted by the Echogarden API and CLI.
+
+Related resources you may need:
+* [A comprehensive list of all supported engines](Engines.md)
+* [A guide on how to use the command line interface](CLI.md)
 
 ## Synthesis
 

package/docs/Tasklist.md

@@ -0,0 +1,289 @@
+# Developer's task list
+
+## Bugs
+
+### Audio player
+* In rare situations, the audio player fails when encountering encoded markup (it can't find it in the text). Decide what to do when this happens
+
+### Phoneme processing
+* IPA -> Kirshenbaum translation is still not completely similar to what is output by eSpeak. Also, in rare situations, it outputs characters that are not accepted by eSpeak and eSpeak errors. Investigate when that happens and how to improve on this
+
+### Segmentation
+* eSpeak workaround for getting markers to work after sentence boundaries fails on some edge cases, especially when the input has special characters. Sequences like `**********` or `----------` fail.
+
+### Browser extension
+* Investigate why WebSpeech events sometimes completely stop working in the middle of an utterance for no apparent reason. Sometimes this is permanent, until the extension is restarted. Is this a browser issue?
+* If a request is made and the server takes too much time to respond, the service worker may sleep and the request never canceled
+
+### Browser extension / content script
+* Highlighting sometimes does not appear when mouse is pressed over handle while speech of element starts
+
+## Features and enhancements
+
+### Browser extension
+* Options UI
+* Add supported engines and voices to WebSpeech voice list
+* Pause and resume support
+
+### Browser extension / content script
+* Autoscroll should work even if the scrollbar relevant to the target element is not the viewport's scrollbar
+* Find a way to show handles even for elements that start with a link
+* Add detection for line breaks in `pre` blocks
+* Some symbols, like `=`, `*`, `#` are not highlighted
+* Support the custom tags used in YouTube comments
+* Show handles based on `<br>` tags and possibly line breaks internal to the element
+* Show handles based on sentence start positions
+* UI or gesture to stop speech (other than the `esc` key)
+* Hide handles when mouse leaves the viewport
+* Don't show handles when mouse is over a large container element
+* Button or keyboard shortcut to show and hide handles
+* Show blinking placeholder when synthesis is loading for a particular text node
+* Navigate paragraphs or sentences with keyboard shortcuts
+* Minimum size when iterating text nodes to get handle
+
+### Worker
+* Optionally omit unnecessary data from the response (decoded input, segment data, etc.)
+* Support compressed audio in response
+* Add cancelation checks in more operations
+* Support more operations
+
+### CLI
+* Don't show an error when `--help` option is given, instead suggest to type `echogarden` or `echogarden help` to get help
+* Find a way to ensure that a user who typed `align audio.mp3 transcript.txt` and then changed to `transcribe audio.mp3 transcript.txt` won't accidently overwrite their transcript file. Simple solution, but possibly not the best solution: `align audio.mp3 --reference=transcript.txt`. Other solution: on `transcribe` and `translate-speech`, ask if output file already exist or require an `--overwrite` flag to ensure that the user intended to overwrite the existing file.
+* Restrict input media file extensions to a set list to avoid cases where a media file would be overwritten due to user error
+* Show a message when a new version is available
+* Figure out which terminal outputs should go to stdout, or if that's a good idea at all
+* Option to set audio output codec options
+* Option to set audio output device
+* Print available synthesis voices when no voice matches (or suggest near matches)
+* `transcribe` can also accept `http://` and `https://` URLs
+* Use a file type detector like `file-type` that uses magic numbers to detect the type of a binary file regardless of its extension. This would help giving better error messages when the given file type is wrong.
+* Consider adding the text offset to each segment, sentence and word in the resulting timeline with respect to the original file (even if it is, say, HTML or captions file)
+* Add phone playback support
+* More fine-grained intermediate progress report for operations
+* Suggest possible correction on the error of not using `=`, e.g. `speed 0.9` instead of `speed=0.9`
+* Multiple configuration files in `--config=..` taking precedence by order
+* Support comments in JSON configuration file
+* Generate JSON configuration file schema
+* Make enum options case-insensitive if possible
+* Mode to print IPA words when speaking
+* Support list-typed properties configuration files (already supported in JSON)
+
+### CLI / `speak-wikipedia`
+* Correctly detect language when a Wikipedia URL is passed instead of an article name
+* Add option to set language edition separately from language, since they use different language codes in some cases
+
+### CLI / `speak-url`
+* Use the Wikipedia reader when the URL is detected to be from `wikipedia.org`
+
+### CLI / `list-packages`
+* Support filters
+
+### CLI / Configuration file
+* Support arrays
+
+### CLI / New commands
+* `help`: Show help for a particular command, like `help transcribe`
+* `list-engines`: List available engines for a particular command, like `list-engines speak`
+* `play-with-captions`: Preview captions in terminal
+* `play-with-timeline`: Preview timeline in terminal
+* `captions-to-text`, `captions-to-timeline`, `srt-to-vtt`, `vtt-to-srt`
+* `crop-to-timeline`, `split-by-timeline`
+* `text-to-ipa`, `arpabet-to-ipa`, `ipa-to-arpabet`
+* `phonemize-text`
+* `normalize-text`
+* `pos-tag-text`
+* `remove-nonspeech`
+* `speak-youtube`: To speak the transcript of a YouTube video
+
+### API
+* Option to control logging verbosity
+* Add support to accept caption options in API and CLI
+* Support full language names as inputs
+* Retry on error when connecting to cloud providers, including WebSocket disconnection with `microsoft-edge` (already supported by `gaxios`, not sure about `ws` - decide on default setting)
+* Validate timelines to ensure timestamps are always increasing, no -1 timestamps or timestamps over the time of the audio, no sentences without words, etc. and correct if needed
+* Time/pitch shifting for recognition and alignment results
+* Add support for phrases in timelines
+* Accept voice list caching options in `SynthesisOptions`
+
+### Language detection
+* Deploy and add the new language detection model
+* When using Whisper for language detection of speech, apply it to the entire audio, not just the first 30 seconds
+
+### Segmentation
+* Option to split segment on single line break as well as double line break (when splitting on double line breaks, there might be cases where a single line break should be seen as a sentence boundary - a line in song lyrics). Maybe a better approach is to optionally preprocess the text and merge subsequent lines than are intended to be a part of the same paragraph. In this way, paragraph breaks would always be single line breaks, and there's no need to carry settings for this detail within the program.
+* Split long words
+* See if it's possible to reliably use eSpeak as a segmentation engine
+* Path to `kuromoji` dictionaries can be found more reliably than current
+
+### Captions
+* If a subtitle is too short and at the end of the audio, try to extend it back if possible (for example, if the previous subtitle is already extended, take back from it)
+* Split long words if needed
+* Decide how many punctuation characters to allow before breaking to a new line (currently it's infinite)
+* Add more clause separators, for even more special cases
+* Add option to output word or phoneme-level caption files (investigate how it's done on YouTube auto-captions)
+* Parse VTT's language
+* Option to generate captions that have word-level timings
+
+### Synthesis
+* Find places to add commas (",") to improve speech fluency. VITS voices don't normally add phrasing breaks if there is no punctuation
+* An isolated dash " - " can be converted to a " , " to ensure there's a break in the speech.
+* Ensure abbreviations like "Ph.d" or names like are segmented and read correctly (why doesn't `cldr` treat it as a word? Maybe it's not getting the right parameters, or it's not included in the list?) and "C#"
+* Use preprocessed eSpeak in places other than VITS
+* Way to manually reset voice list cache
+* When synthesized text isn't pre-split to sentences, apply sentence splits by using the existing method to convert the output of word timelines to sentence/segment timelines
+* Log full language of selected voice (it may have a different dialect than expected)
+* Add partial SSML support for all engines. In particular, allow changing language or voice using the `<voice>` and `<lang>` tags, `<say-as>` and `<phoneme>` where possible.
+* Some `sapi` voices and `msspeech` languages output phones that are converted to Microsoft alphabet, not IPA symbols. Try to see if these can be translated to IPA
+* Decide whether asterisk `*` should be spoken when using `speak-url` or `speak-wikipedia`
+* Decide what to do with `«` and `»` punctuation characters (guillemets) when parsing and playing
+* Use VAD on the synthesized audio file to get more accurate sentence or word segmentation
+* Try to remove reliance on `()` after `.` character hack in `EspeakTTS.synthesizeFragments`.
+* eSpeak IPA output puts stress marks on vowels, not syllables - which is the standard for IPA. Consider how to make a conversion to and from these two approaches (possibly detect it automatically).
+* Investigate if `espeak` can be made to correctly support phonemizing and pronouncing the dot character like in `object.key`
+* Speaker-specific voice option
+* Decide if `msspeech` engine should be selected if available. This would require attempting to load a matching voice, and falling back if it is not installed
+* Option to disable alignment
+
+### Synthesis / preprocessing
+* Extend the heteronyms JSON document with more words
+* Full date normalization (e.g. `21 August 2023`)
+* Support custom lexicons. For example, a lexicon for general pronunciation corrections (for example `ee-lon` instead of `eh-lon`)
+* Add support for capitalized only rules, and possibly also all uppercase / all lowercase rules.
+* Support normalizing to graphemes, not only phonemes
+* Is it possible to pre-phonemize common words like "the" or is it a bad idea / not necessary?
+* Add support for user-defined lexicons for VITS synthesis (or any other one that supports preprocessing)
+* Add support for text normalization preprocessing for all engines that can benefit from it (possibly including cloud engines).
+* Add SAPI pronunciation to lexicons (the information is already there for `en_US` and `en_GB`)
+* Try to use entity recognition to detect years, dates, currencies etc., which would disambiguate cases where it is not clear, like "in 1993" in "She was born in 1993" and "It searched in 1993 websites"
+* Option to add POS tags to timeline, if available
+* Cache lexicons to avoid parsing the JSON each time it is loaded (this may not be needed for now since the existing one is relatively small)
+
+### VITS
+* Allow to limit how many models are cached in memory
+* Custom model paths (decide how to implement)
+* Pull voice list from JSON file based on URL? Is that a good idea?
+
+### Recognition
+* Add confidence to each recognized word, if available
+* Show alternatives when playing in the CLI. Clear current line and rewrite already printed text for alternatives during the speech recognition process
+* Look for good split points using VAD before performing recognition
+* Option to split recognized audio to segments or sentences, as is done with synthesized audio
+
+### Whisper
+* When using `dtw-ra` alignment, pass the transcript as a prompt. Remove some of the initial transcript based on what has been detected (try to find the best matching initial segment between the transcript and recognized text, and remove it at each recognition window).
+* During language detection, if file is more than 30s, run the detection over all the segments and average the resulting probability distributions, consider how to handle very short segments
+* Timestamps extracted from cross-attention are still not as accurate as what the official Python implementation gets. Try to see if you can make them better.
+* Cache last model
+* Integrate speech language detection into the recognition itself, so it is done efficiently when the language is not known
+* Bring back option to use eSpeak DTW based alignment on segments, as an alternative approach
+* The segment output can be use to split to segment files, otherwise it is possible to try to guess using the pause lengths or voice activity detection
+* Way to specify model size, such that the English-only/multilingual would be auto selection for sizes other than `tiny`?
+
+### Postprocessing
+* When `normalize` is set to false, should obvious clipping still be prevented?
+
+## Maintenance and cleanup
+
+* Find a way to reset voice list cache on update
+* CLI code has a lot of repetition. See how it can be refactored
+* See if the installation of `winax` can be automated and only initiate if it is in a Windows environment
+* Ensure that all modules have no internal state other than caching
+* Start thinking about some modules being available in the browser. Which node core APIs the use? Which of them can be polyfilled, an which cannot?
+* Change all the Emscripten WASM modules to use the `EXPORT_ES6=1` flag to all of them and rebuild them. Support for node.js was only added in September 2022 (https://github.com/emscripten-core/emscripten/pull/17915), so maybe wait a little bit until it is stable.
+* Remove built-in voices from `flite` to reduce size?
+
+## External bugs
+
+* `espeak-ng`: 'Oh dear!”' is read as "oh dear exclamation mark", because of the special quote following the exclamation mark
+* `espeak-ng`: [Marker right after sentence end is not reported as an event](https://github.com/espeak-ng/espeak-ng/issues/920)
+* `espeak-ng`: On Japanese text, it says "Chinese character" or "Japanese character" for characters it doesn't know
+* `wtf_wikipedia` Sometimes fails on `getResult.js` without throwing a humanly readable error
+* `wtf_wikipedia` Sometimes captures markup like `.svg` etc.
+* `msspeech`: Initialization fails on Chinese and Japanese voices (but not Korean)
+* `compromise`: Slow initialization time. Currently it takes more than a second
+* Chromium doesn't fire timer events when cursor is positioned over scrollbar or body margins
+
+## Things to test
+
+* Test that SSML works where it should
+* Test synthesis, recognition and alignment with empty input. Do they still work?
+
+## Future features and enhancements
+
+### CLI
+* Synthesize given subtitle file and try to preserve the existing timing of cues, or even align to existing speech
+* Auto-generate options file, with comments, based on default options of the API
+* Have the CLI launch a background worker (in a thread) to enable better parallelism
+* Play back result audio while synthesis or recognition is still processing on the background (may require `worker_threads`)
+* Navigate up down backward forward on file with timeline
+* Auto-import project Gutenberg texts (by URL or from a file)
+* `stdin` input support
+* `stdout` output support
+* Markdown file as text input?
+
+### API
+* Auto-install npm modules when needed using something like `npm-programmatic`
+
+### Text enhancement
+* Add capitalization and punctuation when to recognition outputs (Silero has a model for it for `en`, `de`, `ru`, `es`, but in `.pt` format only)
+
+### Synthesis / preprocessing
+* Extend preprocessing to other language versions of `compromise`. There are versions for French, German, Italian and Spanish
+
+### Recognition
+* Low latency recognition mode. Make the partial transcription available as fast as possible
+* Live input / microphone recognition
+* Live vosk alternatives events
+* Implement beam search for Whisper decoder
+* Implement beam search for Silero decoder
+
+### Web
+* Web based frontend UI to the server
+* Adapt some WASM modules to also run on the web
+* Investigate running in WebContainer
+
+### Alignment
+* Align audio file to audio file
+* Alignment with speech translation assistance, which would enable multilingual subtitle replacement for translated captions
+* Make `dtw` mode work with more speech synthesizers to produce its reference
+
+## Documentation
+
+### CLI
+* Document the `serve` command
+
+## Possible new engines or platforms
+
+* OpenAI Whisper cloud service (`large-v2` model is available, at a price).
+* [Assembly AI cloud service](https://www.assemblyai.com/)
+* [Deepgram cloud service](https://deepgram.com/)
+* `whisper.cpp` CLI and WASM support
+* Coqui STT server connection
+* See what can be done on supporting WinRT speech: in particular `windows.media.speechsynthesis` and `windows.media.speechrecognition` support, possibly using NodeRT or some other method.
+* Figure out how to support `julius` speech recognition via WASM.
+* Any way to support RHVoice?
+* Silero text enhancement engine can be ported to ONNX
+* Investigate Raspberry Pi support. In particular, see if `onnxruntime-node` can be built for this environment
+* Reimplement KNN model in ONNX for better performance
+
+## Maybe?
+
+* PDF support
+* Using a machine translation model to provide speech translation to languages other than English?
+* Is it possible to get sentence boundaries without punctuation using NLP techniques like part of speech tagging?
+
+## May or may not be good ideas
+
+* Bring back interleaved playback
+* Bring back debugging file output
+
+## Other ideas
+
+* HTML generator, that includes text and audio, with playback and word highlighting
+* Video generator
+* Desktop app that uses the tool to transcribe the PC audio output
+* Special method to use time stretching to project between different utterances of the same text
+* Is it possible to combine the Silero speech recognizer and a language model and try to perform Viterbi decoding to find alignments?
+* Voice replacement
+* Predict timing for individual letters (graphemes) based on phoneme timestamps

package/docs/Technical.md

@@ -2,7 +2,7 @@
 
 * Echogarden is written in TypeScript and targets the Node.js platform.
 * It uses ESM modules and latest ECMAScript and TypeScript features.
-* It does not depend on essential binary executables. Instead, all of its engines either use pure JavaScript, WebAssembly, WASI, or the ONNX runtime (with some exceptions: the CLI does invoke a few binary executables, loaded from expansion packages, for the `SoX` and `ffmpeg` tools.Using expansion packages simplifies the installation and ensures non-buggy version are used. Since SoX `v14.4.2` is broken on Windows, it bundles `v14.4.1`).
+* It does not depend on essential binary executables. Instead, all of its engines either use pure JavaScript, WebAssembly, WASI, or the ONNX runtime, with some exceptions: the CLI does invoke a few binary executables, loaded from expansion packages, for the `SoX` and `ffmpeg` tools. Using expansion packages simplifies the installation and ensures non-buggy version are used. Since SoX `v14.4.2` is broken on Windows, it bundles `v14.4.1`.
 * It does not depend on essential native node.js modules requiring compilation with `node-gyp`. This greatly simplifies the installation experience for end-users (the ONNX runtime bundles precompiled NAPI modules for all supported platforms - it doesn't require any compilation during its installation).
 
 ## Package system
@@ -34,7 +34,7 @@ Currently, the biggest contributors to the size are:
 
 `onnxruntime-node` is big because it bundles pre-compiled binaries for multiple platforms. `kuromoji` is large because of its dictionary files and some unessential test code it bundles. The other three packages include large WASM binaries.
 
-So, yes, in the future it may be possible to reduce the core installed size by dynamically installing some of these dependencies, or using modified, "slimmed-down" custom versions.
+So, yes, in the future it may be possible to reduce the core installed size by dynamically installing some of these dependencies, or using modified, "slimmed-down" versions of some packages.
 
 ## Since the code is almost all JavaScript and WASM, why can't it just run in a web browser?
 

package/package.json

@@ -1,21 +1,19 @@
 {
 	"name": "echogarden",
-	"version": "0.5.0",
+	"version": "0.5.2",
 	"description": "A fully open-source speech system designed with end-users in mind.",
 	"author": "Rotem Dan",
 	"license": "GPL-3.0-only",
 	"keywords": [
-		"tts",
 		"text-to-speech",
-		"stt",
-		"speech-to-text",
-		"speech processing",
 		"speech synthesis",
+		"speech-to-text",
 		"speech recognition",
+		"speech processing",
 		"speech alignment",
 		"forced alignment",
 		"speech translation",
-		"language recognition",
+		"language identification",
 		"language detection"
 	],
 	"repository": {

package/docs/Roadmap.md

@@ -1,15 +0,0 @@
-# Roadmap
-
-- [x] Publish a working command line application
-- [x] Smooth out any significant bugs
-- [x] Ensure platform support
-- [x] Add missing features, options or commands to the CLI
-- [x] Design a message-based worker API
-- [x] Implement the worker API
-- [x] Implement a WebSocket server that implements the worker API
-- [ ] Publish the worker API. Add any extra methods or options needed for that to happen
-- [ ] Develop a browser extension, talking the WebSocket server
-- [ ] Develop a web-based client app, talking to, and hosted on the server
-- [ ] Modify the CLI app to internally use the background worker for some tasks
-- [ ] Further improve the client app
-- [ ] Port some of the offline speech synthesis API to the browser (in particular the `vits`, `pico`, `flite`, `eSpeak` engines)