echogarden 0.12.2 → 1.0.0

package/docs/Releases.md

@@ -0,0 +1,262 @@
+# Release notes
+
+## `1.0.0` (April 4, 2024)
+
+**New features**:
+* Add support for [`whisper.cpp`](https://github.com/ggerganov/whisper.cpp), a C++ port of OpenAI's Whisper speech recognition architecture. It is faster than the integrated `whisper` engine, supports large models, and GPU processing. It can now be used for recognition, speech translation and alignment (via `dtw-ra`), though its word timestamps are less accurate than the integrated `whisper` engine.
+* Add the [MDX-NET](https://github.com/kuielab/mdx-net/) source separation model, enabling vocal tracks to be extracted from music and speech audio using the new `isolate` operation
+* Add support for OpenAI cloud platform's [speech recognition and translation services](https://platform.openai.com/docs/guides/speech-to-text)
+* Add support for OpenAI cloud platform's [speech synthesis service](https://platform.openai.com/docs/guides/text-to-speech)
+* Integrate optional vocal isolation to speech recognition, alignment and translation operations using the new `--isolate` option, allowing for higher accuracy in difficult cases like achieving word-level lyrics alignment
+* Add the new `adaptive-gate` VAD engine using a custom bandlimited adaptive gate. Fast and robust. Works well for relatively clean tracks or tracks that have already been processed using vocal isolation
+* Add optional token-level repetition suppression to Whisper engine
+* Expose several new configuration options for the Whisper engine: a settings for maximum tokens per part, and a setting to enable/disable repetition suppression, set custom random seed, disable/enable decoding of timestamp tokens
+* Expose more options for the Elevenlabs engine
+
+**Behavioral and breaking changes**:
+* Minimal required node version changed to `18.0.0`
+* All recognition, alignment, translation and language recognition operations first apply the new adaptive gate VAD (can be changed to any other VAD engine via the `vad.` option prefix) and remove any sections that are not identified as containing voice, before starting processing. This should improve results in most cases, and reduce processing time
+* To reduce Whisper hallucinations and repetition loops, these change were made:
+	* Pre-cropping by default can significantly help with reducing hallucinations, giving the model less "empty space" to hallucinate on
+	* Enable new token-level repetition suppression (`suppressRepetition = true`) during decoding
+	* Disable decoding of timestamp tokens by default (`decodeTimestampTokens = false`), since more accurate timing is already extracted via cross-attention weight alignment. For unclear reasons, this can significantly reduce the occurrence of token repetition loops, and increases word timestamp accuracy. However, there are cases where this causes the model to end a part prematurely, especially in singing and less speech-like voice segments. In those cases the it can be enabled with `decodeTimestampTokens = true`
+* When `transcribe`, `align` or `translate-speech` operations are run with `--isolate` enabled, they will output the isolated part in `some-output-file.isolated.wav` and background part (isolated subtracted from original) in `some-output-file.background.wav` (any supported codec other than `wav` can be used - this is just an example)
+* VAD operations now return a timeline including only the active sections, labeled as `active`
+* When the specified language is not English, but an `.en` Whisper model was specified, a warning would be shown, and the model would be automatically switched to the corresponding multilingual model (omitting the `.en`), instead of producing an error
+* Default speech language detection engine is now changed to `whisper`
+* Default voice activity detection engine is now changed to `silero`
+* CLI: duplicate file name outputs would now append the `_001` suffix pattern instead of ` (1)`. This change is meant to simplify sorting and typing the resulting file names and remove the space and parenthesis characters, to ensure compatibility with all operating systems
+* `targetPeakDb` post-processing and denoising option renamed to `targetPeak`
+* `maxIncreaseDb` post-processing and denoising option renamed to `maxGainIncrease`
+* `dryMixGainDb` denoising option renamed to `dryMixGain`
+
+**Enhancements**:
+* `detect-voice-activity` now also outputs the cropped voice when an audio file is given as output, with `.cropped` suffix added
+* Improved word segmentation of whisper engine. Words spanning multiple recognized time segments are now split. Word timestamp ranges don't overlap non-speech sections.
+* Whisper and Whisper.cpp outputs now include timestamps for individual recognized tokens
+
+**Fixes**:
+* Integrated `whisper` engine now uses `tiktoken` to tokenize text, which produces near identical tokenization compared to the official Python implementation. This fixes issues with decoding Chinese characters, correctly encoding prompts, and several others languages with characters that span multiple tokens
+* Fix voice language lists for multilingual voices in Elevenlabs TTS engine
+* Fix runtime error with `rnnoise` when used as VAD engine
+* Fix issue in CLI where supported output media formats weren't reported correctly
+* Fix issue with identifying legacy IEEE Float wave format
+* Fix issue with language detection failing with empty inputs
+* Fix issue with Whisper model failing when no tokens are detected in a part
+* Fix issue with phone alignment not getting the right DTW window size, producing warnings like `all cost directions are equal to infinity
+* Fix phone timelines in DTW-RA
+* Add more workarounds for eSpeak tokenization bugs
+* Various other fixes
+
+**Documentation**:
+* Add new 'Releases' page. Releases before `1.0.x` were retroactively documented based on commit history (may not exactly detail all historical changes)
+* Options reference page restructured and updated with some missing information
+
+## `0.12.x` (March 16, 2024)
+
+**New features**:
+* Add global options support to API
+* Add updated `ffmpeg` internal packages for many platforms, including pre-signed binaries for macOS
+
+**Enhancements**:
+
+**Fixes**:
+* Fix issue with Polish TTS in eSpeak NG
+* Fix warning about `punycode` module in the CLI
+
+**Other**:
+* Include `package-lock.json` in repository
+
+## `0.11.x` (August 17, 2023)
+
+Many features, enhancements, and fixes were incrementally added over the span of 7 months, up to March 2024.
+
+**New features**:
+* Partial rewrite of subtitle generation methods. Adds many features and options
+* Expose subtitle methods to API
+* Add support for multi-pass (AKA hierarchical) DTW alignment
+* Auto-prompt parts in `whisper` STT engine
+* Add support for Klatt synthesis in `espeak` engine
+
+**Enhancements**:
+* Adds a total of 14 new VITS voices
+* Retry on failure for `microsoft-edge` and `google-translate` TTS engines
+
+**Fixes**:
+* Don't error when empty audio is returned in `microsoft-edge` response
+* Fix audio playback in macOS
+* Many other fixes
+
+## `0.10.x` (August 2, 2023)
+
+**New features**:
+* Add developer and server APIs, allowing the package to be used as a library or server
+* Add granularity options to DTW
+* Add Linux SoX package
+
+**Enhancements**:
+* Adds 7 new VITS voices
+
+**Fixes**:
+* Many fixes
+
+## `0.9.x` (July 29, 2023)
+
+**New features**:
+* Add text offsets to timeline
+* Include segments and sentences in recognition and speech translation timelines
+* Improve voice information in Elevenlabs engine
+
+**Behavioral changes**:
+* Set `en_GB-alan-low` as default `en-GB` VITS voice, since `danny` was trained to accept `en-US` pronunciations, which may confuse some people
+
+**Fixes**:
+* Work around several eSpeak bugs
+* Many fixes
+
+## `0.8.x` (July 25, 2023)
+
+**New features**:
+* Add confidence to some speech recognition timelines
+* Add `plaintext` options
+* Add punctuation thresholds to Whisper decoder
+* Expose subtitle configuration options to CLI
+
+**Behavioral changes**:
+* Rename `subtitles.minWords` to `subtitles.minWordsInLine`
+
+**Enhancements**:
+* Improve speech language detection to work with arbitrary length audio. Split audio to overlapping parts and detect each part individually. Then average the results.
+* Add more heteronyms
+
+**Fixes**:
+* Improve numerical stability of softmax function
+* Work around eSpeak bug with markers in long inputs
+* Various fixes
+
+## `0.7.x` (July 23, 2023)
+
+**New features**:
+* Alignment: Add option to accept custom lexicons
+* Implement and add decoder temperature option to Whisper model
+
+**Behavioral changes**:
+
+**Enhancements**:
+* Alignment: use preprocessing and lexicons
+* Add language detection to speech translation
+* Add language detection options to synthesis, and set empty default options for detection in several APIs
+* Accept speech language detection options in recognition
+* Show warning only when DTW window is smaller than 25% of audio duration
+* Add colors to log messages
+* Warn when maximum DTW window duration is smaller than source audio duration.
+
+**Fixes**:
+* Fix support for SSML input in eSpeak engine
+* Various fixes
+
+## `0.6.x` (July 20, 2023)
+
+**New features**:
+* Add support for custom lexicons. Change lexicon object structure to include language code and allow for multiple languages in a single lexicon
+* Add support for SSML inputs (currently only supported by Google, Microsoft and Amazon cloud engines). Ensure they are not split to segments or sentences
+* CLI: Add flag to enable or disable file overwriting in CLI
+* CLI: Add command to list engines
+* Add arguments to customize paragraph parsing
+* Change whitespace option to include option for collapsing all whitespace
+
+**Behavioral changes**:
+* Set Elevenlabs defaults to mid-values
+* Split plain text to paragraphs using double line breaks by default.
+* Rename `awsPolly` options to `amazonPolly` to be more consistent with documentation
+
+**Enhancements**:
+* Change whitespace option to include option for collapsing all whitespace
+* Add awareness of guillemets
+* Log full language of selected voice
+
+**Fixes**:
+* Fix and update Elevenlabs engine
+* Fix incorrect gender properties for some VITS voices
+* Convert to plaintext before detecting language when input is SSML.
+* Various fixes
+
+**Other**:
+* Remove `package-lock.json` from the repository
+
+## `0.5.x` (July, 19 2023)
+
+**New features**:
+
+**Behavioral changes**:
+* Move to new package system supporting version tags, and hosted in a Hugging Face repository
+
+**Enhancements**:
+* Add large numbers of VITS voices
+
+**Fixes**:
+* Various fixes
+
+## `0.4.x` (July 9, 2023)
+
+**New features**:
+
+**Behavioral changes**:
+* Remove support for `afplay` and `aplay` for playback. Only SoX is used now.
+
+**Enhancements**:
+* Improve text normalization and add support for currencies
+* Improve heteronym lexicons
+* Extend year patterns
+* Improve logic for decade normalization
+* Add some British English (RP) pronunciations to heteronym lexicon
+* Add large numbers of VITS voices
+
+**Fixes**:
+* Various fixes
+
+## `0.3.x` (July 1, 2023)
+
+**New features**:
+
+**Behavioral changes**:
+
+**Enhancements**:
+* Upgrade heteronym disambiguation to an improved, rule-based approach, which doesn't use POS tagging
+* Extend decade normalization
+* Improve text normalization and add support to currencies
+* Update the heteronym lexicon
+
+**Fixes**:
+* Remove sentences containing only whitespace when synthesizing
+* Many fixes
+
+## `0.2.x` (May 10, 2023)
+
+**New features**:
+
+**Behavioral changes**:
+* Remove dependency on `xregexp` package. Use Unicode RegExp instead.
+
+**Enhancements**:
+* Show current sentence and segment in synthesis log.
+* Add check for cancellation flag.
+* Add support to additional VITS voices
+* Improve auto TTS engine selection
+* Many Enhancements
+
+**Fixes**:
+* Many fixes
+
+
+## `0.1.x` (April 24, 2023)
+
+Initial release
+
+**Enhancements**:
+* Add support for skipping audio playback with the Enter key.
+* Run CLI in a worker thread by default.
+* Many Enhancements
+
+**Fixes**:
+* Many fixes

package/docs/Server.md

@@ -15,13 +15,13 @@ echogarden serve [options]
 * `keyPath`: Path to a private key file, required when `secure = true`
 * `deflate`: Use per-message deflate. Defaults to `true`
 * `maxPayload`: Maximum raw message payload size (in bytes). Defaults to `1000 * 1000000` (1GB)
-* `useWorkerThread`: Run worker in a separate thread. Defaults to `true` (recommended to leave as is)
+* `useWorkerThread`: Run worker in a separate thread. Defaults to `true` (recommended leaving as is)
 
 ## Using the client class
 
 For Node.js clients, a simple client class allows to wrap communications with the server in a more convenient interface, without needing to know the details of the protocol.
 
-Currently, the client is is embedded in the main codebase. This means you have to import the `echogarden` package to use it:
+Currently, the client is embedded in the main codebase. This means you have to import the `echogarden` package to use it:
 
 ```ts
 import { WebSocket } from 'ws'
@@ -39,7 +39,7 @@ ws.on("open", async () => {
 In the future, this module may be separated to an independent lightweight package.
 
 **TODO**: Document using the client class with a background worker.
-**TODO**: Add support for cancelation signals in the client class.
+**TODO**: Add support for cancellation signals in the client class.
 
 ## Protocol details
 
@@ -111,18 +111,18 @@ Example response, for the above synthesis request:
 }
 ```
 
-### Cancelation messages
+### Cancellation messages
 
-To cancel an existing request, the client can send a `CancelationRequest` message, with the same `requestId` of an ongoing request, like:
+To cancel an existing request, the client can send a `CancellationRequest` message, with the same `requestId` of an ongoing request, like:
 
 ```ts
 {
-	messageType: 'CancelationRequest',
+	messageType: 'CancellationRequest',
 	requestId: 'cb7e0f3ec835a213b005c4424c8d5775'
 }
 ```
 
-**TODO**: Cancelation requests are currently only supported for synthesis operations. Extend support for other operations.
+**TODO**: Cancellation requests are currently only supported for synthesis operations. Extend support for other operations.
 
 ## Starting the server programmatically
 

package/docs/Tasklist.md

@@ -2,6 +2,9 @@
 
 ## Bugs
 
+### Synthesis
+
+
 ### Phoneme processing
 * IPA -> Kirshenbaum translation is still not completely similar to what is output by eSpeak. Also, in rare situations, it outputs characters that are not accepted by eSpeak and eSpeak errors. Investigate when that happens and how to improve on this.
 
@@ -12,54 +15,49 @@
 ### Browser extension / content script
 * Highlighting sometimes does not appear when mouse is pressed over handle while speech of element starts
 
-## Features and enhancements
+### External bugs
 
-### Browser extension
-* Options UI
-* Add supported engines and voices to WebSpeech voice list
-* Pause and resume support
+* `espeak-ng`: 'Oh dear!”' is read as "oh dear exclamation mark", because of the special quote character following the exclamation mark
+* `espeak-ng`: [Marker right after sentence end is not reported as an event](https://github.com/espeak-ng/espeak-ng/issues/920)
+* `espeak-ng`: On Japanese text, it says "Chinese character" or "Japanese character" for characters it doesn't know
+* `wtf_wikipedia` Sometimes fails on `getResult.js` without throwing a humanly readable error
+* `wtf_wikipedia` Sometimes captures markup like `.svg` etc.
+* `msspeech`: Initialization fails on Chinese and Japanese voices (but not Korean)
+* `compromise`: Slow initialization time. Currently, it takes more than a second
+* Chromium doesn't fire timer events when cursor is positioned over scrollbar or body margins
+* `whisper.cpp`: Timestamps aren't very accurate when `enableDTW` is set. There's a constant lag
+* Node.js WASI for `flite` on Node `v21.7.2` and `v20.12.1` is intermittently crashing the process when `run` is called
 
-### Browser extension / content script
-* Autoscroll should work even if the scrollbar relevant to the target element is not the viewport's scrollbar
-* Find a way to show handles even for elements that start with a link
-* Add detection for line breaks in `pre` blocks
-* Support the custom tags used in YouTube comments
-* Show handles based on `<br>` tags and possibly line breaks internal to the element
-* Show handles based on sentence start positions
-* UI or gesture to stop speech (other than the `esc` key)
-* Hide handles when mouse leaves the viewport
-* Don't show handles when mouse is over a large container element
-* Button or keyboard shortcut to show and hide handles
-* Show blinking placeholder when synthesis is loading for a particular text node
-* Navigate paragraphs or sentences with keyboard shortcuts
-* Minimum size when iterating text nodes to get handle
+## Features and enhancements
 
 ### Server
-* Option to allow or disallow local file paths as arguments to API methods (as a security guard)
+* Option to allow or disallow local file paths as arguments to API methods (as a security safeguard)
 
 ### Worker
-* Add cancelation checks in more operations
+* Add cancellation checks in more operations
 * Support more operations
 
 ### CLI
-* Show names of files written do disk. This is useful for cases where a file is auto-renamed to allow overwrite.
-* Consider if auto-renamed files can use a suffix other than ` (1)`.
-* Restrict input media file extensions to ensure that invalid files are not passed to FFMpeg.
+* Show names of files written do disk. This is useful for cases where a file is auto-renamed to prevent overwriting existing data
+* Restrict input media file extensions to ensure that invalid files are not passed to FFmpeg
 * Mode to print IPA words when speaking
 * Consider what to do with non-supported templates like `[hello]`
 * Show a message when a new version is available
 * Figure out which terminal outputs should go to stdout, or if that's a good idea at all
-* Option to set audio output device for playback
 * Print available synthesis voices when no voice matches (or suggest near matches)
 * `transcribe` may also accept `http://` and `https://` URLs and pull the remote media file
-* Make enum options case-insensitive if possible
-* Add phone playback support
+* Make `enum` options case-insensitive if possible
 * More fine-grained intermediate progress report for operations
 * Suggest possible correction on the error of not using `=`, e.g. `speed 0.9` instead of `speed=0.9`
 * Multiple configuration files in `--config=..` taking precedence by order
 * Generate JSON configuration file schema
-* Use a file type detector like `file-type` that uses magic numbers to detect the type of a binary file regardless of its extension. This would help giving better error messages when the given file type is wrong.
-* Option to set playback volume.
+* Use a file type detector like `file-type` that uses magic numbers to detect the type of a binary file regardless of its extension. This would help to give better error messages when the given file type is wrong.
+
+### CLI / playback
+* Option to set audio output device for playback
+* Option to set playback volume
+* Maybe find a way not to pre-normalize if the audio is silent (to prevent a 30dB increase of possible noise)
+* Add phone playback support
 
 ### CLI / `speak`
 * Add support for sentence templates, like `echogarden speak-file text.txt /parts/[sentence].wav`.
@@ -72,33 +70,32 @@
 * Use the Wikipedia reader when the URL is detected to be from `wikipedia.org`
 
 ### CLI / `list-voices`
-* When given a configuration file, see if you can fall back to take options from from `speak` options, for example, to take API keys that are required for the both the synthesis request and voice list request and
+* When given a configuration file, see if you can fall back to take options from `speak` options, for example, to take API keys that are required for the both the synthesis request and voice list request and
 
 ### CLI / `list-packages`
 * Support filters
 
 ### CLI / New commands
-* `speak-youtube`: To speak the subtitles of a YouTube video
 * `play-with-subtitles`: Preview subtitles in terminal
 * `play-with-timeline`: Preview timeline in terminal
 * `subtitles-to-text`, `subtitles-to-timeline`, `srt-to-vtt`, `vtt-to-srt`
-* `crop-to-timeline`, `split-by-timeline`
 * `text-to-ipa`, `arpabet-to-ipa`, `ipa-to-arpabet`
 * `phonemize-text`
 * `normalize-text`
-* `remove-nonspeech`
+* `transcribe-youtube`: Transcribe the audio in a YouTube video (requires fetching the audio somehow - which can't be done using the normal YouTube API)
+* `speak-youtube-subtitles`: To speak the subtitles of a YouTube video
 
 ### API
 * Option to control logging verbosity
-* Retry on error when connecting to cloud providers, including WebSocket disconnection with `microsoft-edge` (already supported by `gaxios`, not sure about `ws` - decide on default setting)
 * Accept full language names as language identifiers
-* Validate timelines to ensure timestamps are always increasing, no -1 timestamps or timestamps over the time of the audio, no sentences without words, etc. and correct if needed
-* See if it's possible to detect and include Emoji characters in timelines.
+* Validate timelines to ensure timestamps are always increasing: no negative timestamps or timestamps over the duration of the audio. No sentences without words, etc. and correct if needed
+* See whether it's possible to detect and include / remove Emoji characters in timelines
 * Add support for phrases in timelines
 * Accept voice list caching options in `SynthesisOptions`
 
 ### Package manager
-* Better error message when package is not found remotely. Currently it just gives a `404 not found` without any other information.
+* Better error message when package is not found remotely. Currently, it just gives a `404 not found` without any other information.
+* Retry on network failure
 
 ### Speech language detection
 
@@ -118,10 +115,10 @@
 * Parse VTT's language
 
 ### Synthesis
-* Option to disable alignment (only for some engines). Alternative: use a low setting that is very fast to compute
+* Option to disable alignment (only for some engines). Alternative: use a low granularity setting that is very fast to compute
 * Find places to add commas (",") to improve speech fluency. VITS voices don't normally add speech breaks if there is no punctuation
 * An isolated dash " - " can be converted to a " , " to ensure there's a break in the speech.
-* Ensure abbreviations like "Ph.d" or names like are segmented and read correctly (does `cldr` treat it as a word? Maybe eSpeak doesn't recognize it as a word). "C#" as well
+* Ensure abbreviations like "Ph.d" or names like are segmented and read correctly (does `cldr` treat it as a word? Maybe eSpeak doesn't recognize it as a word). "C#" and ".NET" as well
 * Find way to manually reset voice list cache
 * When synthesized text isn't pre-split to sentences, apply sentence splits by using the existing method to convert the output of word timelines to sentence/segment timelines
 * Some `sapi` voices and `msspeech` languages output phones that are converted to Microsoft alphabet, not IPA symbols. Try to see if these can be translated to IPA
@@ -132,6 +129,7 @@
 * Decide if `msspeech` engine should be selected if available. This would require attempting to load a matching voice, and falling back if it is not installed
 * Speaker-specific voice option
 * Use VAD on the synthesized audio file to get more accurate sentence or word segmentation
+* When `splitToSentences` is set to `false`, the timeline doesn't include proper sentences. Find a way to pass larger sections to the TTS, but still have proper sentences in the timeline.
 
 ### Synthesis / preprocessing
 * Extend the heteronyms JSON document with additional words like "conducts", "survey", "protest", "transport", "abuse", "combat", "combats", "affect", "contest", "detail", "marked", "contrast", "construct", "constructs", "console", "recall", "permit", "permits", "prospect", "prospects", "proceed", "proceeds", "invite", "reject", "deserts", "transcript", "transcripts", "compact", "impact", "impacts"
@@ -141,40 +139,67 @@
 * Support substituting to graphemes in lexicons, not only phonemes
 * Cache lexicons to avoid parsing the JSON each time it is loaded (this may not be needed for if the file is relatively small)
 * Is it possible to pre-phonemize common words like "the" or is it a bad idea / not necessary?
-* Add support for text preprocessing for all engines that can benefit from it (possibly including cloud engines).
+* Add support for text preprocessing for all engines that can benefit from it (possibly including cloud engines)
 * Add SAPI pronunciation to lexicons (you already have the pronunciations for `en_US` and `en_GB`)
 * Try to use entity recognition to detect years, dates, currencies etc., which would disambiguate cases where it is not clear, like "in 1993" in "She was born in 1993" and "It searched in 1993 websites"
 * Option to add POS tags to timeline, if available
 
 ### Synthesis / VITS
-* Allow to limit how many models are cached in memory
+* Allow limiting how many models are cached in memory
 * Custom model paths (decide how to implement)
 * Pull voice list from JSON file, or based on URL? Is that a good idea?
 * Add speaker names to voice list somehow
 
 ### Synthesis / Azure Cognitive Services
-* Currently, when input is set to be SSML, it is wrapped in a `<speak>` tag. Handle the case where the user made their own SSML document wrapped with a `<speak>` tag as well. Currently it may send invalid input to Azure
+* Currently, when input is set to be SSML, it is wrapped in a `<speak>` tag. Handle the case where the user made their own SSML document wrapped with a `<speak>` tag as well. Currently, it may send invalid input to Azure
 
 ### Recognition
-* Add confidence to each recognized word on any engine that supports it
 * Show alternatives when playing in the CLI. Clear current line and rewrite already printed text for alternatives during the speech recognition process
-* Look for good split points using VAD before performing recognition
 * Option to split recognized audio to segments or sentences, as is done with synthesized audio
+* Try to exclude the timing for trailing punctuation tokens in words that contain them. This can help narrow down the end timestamp to cover the word more tightly
+* Recognized word entries that span VAD boundaries may be split
 
 ### Recognition / Whisper
 * May get stuck in a token repeat loop when silence or non-speech segment encountered in audio. Decide what to do
 * Automatically disable using previous section recognized transcript as prompt for the next section when lots of repetition occurred in previous section
 * Cache last model (if enough memory available)
 * Bring back option to use eSpeak DTW based alignment on segments, as an alternative approach
-* The segment output can be use to split to segments, otherwise it is possible to try to guess using pause lengths or voice activity detection
+* The segment output can be used to split to segments, otherwise it is possible to try to guess using pause lengths or voice activity detection
+* Use compression ratios on the decoded tokens of individual segments and discard if too much repetition detected
 * Way to specify model size only, such that the English-only/multilingual variant would be automatically selected for sizes other than `tiny`?
-* Timestamps extracted from cross-attention are still not as accurate as what the official Python implementation gets. Try to see if you can make them better.
+* Timestamps extracted from cross-attention are still not as accurate as what the official Python implementation gets. Try to see if you can make them better
+* Whisper's Chinese output can be split to words in a more accurate way. Consider using a dedicated segmentation library to perform the segmentation in character sequences that have no spaces within them
 
 ### Alignment
-* Investigate a reliable way to cut silent or non-speech sections to improve results.
 
-### Postprocessing
-* When `normalize` is set to false, should obvious clipping still be prevented?
+* Aligned words entries that span VAD boundaries may be split
+
+### Alignment / DTW-RA
+* Remove emojis and other special characters, that are not likely to be pronounced in the speech, from the transcript timeline before it is synthesized. For example Whisper may produce 'note' emojis when it detects singing or music. Pronouncing them reduces the accuracy of the alignment
+* Optional mode to pass Whisper a special vocabulary of tokens that can appear in the transcript. All other tokens would be suppressed
+
+### Alignment / Whisper
+* New mode to decode the transcript tokens in order using a more standard decoding approach (updating the KV cache at each step). This would allow audio inputs longer than 30 seconds. See if this produces better results
+
+### Browser extension
+* Options UI
+* Add supported engines and voices to WebSpeech voice list
+* Pause and resume support
+
+### Browser extension / content script
+* Autoscroll should work even if the scrollbar relevant to the target element is not the viewport's scrollbar
+* Find a way to show handles even for elements that start with a link
+* Add detection for line breaks in `pre` blocks
+* Support the custom tags used in YouTube comments
+* Show handles based on `<br>` tags and possibly line breaks internal to the element
+* Show handles based on sentence start positions
+* UI or gesture to stop speech (other than the `esc` key)
+* Hide handles when mouse leaves the viewport
+* Don't show handles when mouse is over a large container element
+* Button or keyboard shortcut to show and hide handles
+* Show blinking placeholder when synthesis is loading for a particular text node
+* Navigate paragraphs or sentences with keyboard shortcuts
+* Minimum size when iterating text nodes to get handle
 
 ## Maintenance and cleanup
 
@@ -182,21 +207,10 @@
 * CLI code has a lot of repetition. See how it can be refactored
 * See if the installation of `winax` can be automated and only initiate if it is in a Windows environment
 * Ensure that all modules have no internal state other than caching
-* Start thinking about some modules being available in the browser. Which node core APIs the use? Which of them can be polyfilled, an which cannot?
-* Change all the Emscripten WASM modules to use the `EXPORT_ES6=1` flag to all of them and rebuild them. Support for node.js modules was only added in September 2022 (https://github.com/emscripten-core/emscripten/pull/17915), so maybe wait a little bit until it's' stable.
+* Start thinking about some modules being available in the browser. Which node core APIs the use? Which of them can be polyfilled, and which cannot?
+* Change all the Emscripten WASM modules to use the `EXPORT_ES6=1` flag and rebuild them. Support for node.js modules was only added in September 2022 (https://github.com/emscripten-core/emscripten/pull/17915).
 * Remove built-in voices from `flite` to reduce size?
-* Slim down `kuromoji` package to the reduce base install size
-
-## External bugs
-
-* `espeak-ng`: 'Oh dear!”' is read as "oh dear exclamation mark", because of the special quote character following the exclamation mark
-* `espeak-ng`: [Marker right after sentence end is not reported as an event](https://github.com/espeak-ng/espeak-ng/issues/920)
-* `espeak-ng`: On Japanese text, it says "Chinese character" or "Japanese character" for characters it doesn't know
-* `wtf_wikipedia` Sometimes fails on `getResult.js` without throwing a humanly readable error
-* `wtf_wikipedia` Sometimes captures markup like `.svg` etc.
-* `msspeech`: Initialization fails on Chinese and Japanese voices (but not Korean)
-* `compromise`: Slow initialization time. Currently it takes more than a second
-* Chromium doesn't fire timer events when cursor is positioned over scrollbar or body margins
+* Slim down `kuromoji` package to reduce base installation size
 
 ## Things to test
 
@@ -206,6 +220,7 @@
 * Test everything's fine on macOS
 * Test that cloud services all still work correctly, especially with SSML inputs
 
+
 ## Future features and enhancements
 
 ### CLI
@@ -236,35 +251,37 @@
 * Live input / microphone recognition
 * Implement beam search for Whisper decoder
 * Implement beam search for Silero decoder
-* Live vosk alternatives events
+* Live Vosk alternatives events
 * Investigate exporting Whisper models to 16-bit quantized ONNX or a mix of 16-bit and 32-bit
 
 ### Alignment
-* Implement alignment with speech translation assistance, which would enable multilingual subtitle replacement for translated subtitles
+* Implement alignment with speech-to-text translation assistance, which would enable multilingual subtitle replacement for translated subtitles
 * Method to align audio file to audio file
 * Make `dtw` mode work with more speech synthesizers to produce its reference
 * Predict timing for individual letters (graphemes) based on phoneme timestamps
 
-## Documentation
+### Voice activity detection
+
+* Whisper-based VAD. Use Whisper's 'no speech' token to determine if the audio contains speech
+
+### Source separation
+* Option to customize overlap
+* Add more MDX-NET models
 
 ## Possible new engines or platforms
 
-* OpenAI Whisper cloud service (`large-v3` model is available, at a price).
-* OpenAI Text-to-Speech cloud service.
 * [PlayHT](https://play.ht/) speech synthesis cloud service
-* [Assembly AI cloud service](https://www.assemblyai.com/)
-* [Deepgram cloud service](https://deepgram.com/)
-* `whisper.cpp` CLI and WASM support
+* [Deepgram](https://deepgram.com/) cloud text-to-speech API
+* [Assembly AI](https://www.assemblyai.com/) cloud speech recognition API
 * Coqui STT server connection
+* [MarbleNet VAD](https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/Online_Offline_Microphone_VAD_Demo.ipynb), included of the NVIDIA NeMo framework, can be exported to ONNX
+* Silero text enhancement engine can be ported to ONNX
 * See what can be done on supporting WinRT speech: in particular `windows.media.speechsynthesis` and `windows.media.speechrecognition` support, possibly using NodeRT or some other method.
 * Figure out how to support `julius` speech recognition via WASM.
 * Any way to support RHVoice?
-* Silero text enhancement engine can be ported to ONNX
-* Investigate Raspberry Pi support. In particular, see if `onnxruntime-node` can be built for this environment
 
 ## Maybe?
 
-* PDF support
 * Using a machine translation model to provide speech translation to languages other than English?
 * Is it possible to get sentence boundaries without punctuation using NLP techniques like part of speech tagging?
 
@@ -275,10 +292,12 @@
 
 ## Other ideas
 
+* Support alignment of EPUB 3 eBooks with corresponding audiobook
+* Voice cloning
+* Speech to speech voice conversion
+* Speech-to-speech translation (need to find a good model)
 * HTML generator, that includes text and audio, with playback and word highlighting
 * Video generator
 * Desktop app that uses the tool to transcribe the PC audio output
 * Special method to use time stretching to project between different utterances of the same text
 * Is it possible to combine the Silero speech recognizer and a language model and try to perform Viterbi decoding to find alignments?
-* Voice replacement
-* Speech-to-speech translation (need to find a good model)