echogarden 3.3.0 → 3.4.0

package/docs/Server.md

@@ -27,19 +27,22 @@ For Node.js clients, a simple client class allows to wrap communications with th
 Currently, the client is embedded in the main codebase. This means you have to import the `echogarden` package to use it:
 
 ```ts
-import { WebSocket } from 'ws'
 import { Client } from 'echogarden'
+import { WebSocket } from 'ws'
 
 const ws = new WebSocket('ws://localhost:45054')
 
-ws.on("open", async () => {
+ws.on('open', async () => {
 	const client = new Client(ws)
 
-	const { audio } = await client.synthesize("Hello World", { engine: 'espeak' })
+	const { audio } = await client.synthesize('Hello World', {
+		engine: 'espeak',
+	})
 })
 ```
 
 **TODO**:
+
 * Separate the client to an independent, lightweight, Node.js package, with browser compatibility
 * Add support for cancellation signals
 * Document how to use with a background worker
@@ -72,6 +75,7 @@ The `messageType` property is a string representing the operation to perform. Th
 When sending a message, `requestId` should contain a long random string that uniquely identifies your request, like `cb7e0f3ec835a213b005c4424c8d5775`.
 
 For example, this message requests synthesis:
+
 ```ts
 {
 	messageType: 'SynthesisRequest',
@@ -130,16 +134,21 @@ To cancel an existing request, the client can send a `CancellationRequest` messa
 ## Starting the server programmatically
 
 You can use the `startServer` method to start a new server.
+
 ```ts
-async function startServer(serverOptions: ServerOptions, onStarted: (options: ServerOptions) => void)
+async function startServer(
+	serverOptions: ServerOptions,
+	onStarted: (options: ServerOptions) => void,
+)
 ```
 
 Example:
+
 ```ts
 import { startServer } from 'echogarden'
 
 await startServer({ port: 1234 }, () => {
-	console.log("Server is started!")
+	console.log('Server is started!')
 })
 ```
 

package/docs/Tasklist.md

@@ -11,10 +11,12 @@
 * Option to disable single sentence per cue
 
 ### Browser extension
+
 * Investigate why WebSpeech events sometimes completely stop working in the middle of an utterance for no apparent reason. Sometimes this is permanent, until the extension is restarted. Is this a browser issue?
 * If a request is made and the server takes too much time to respond, the service worker may sleep and the request never canceled
 
 ### Browser extension / content script
+
 * Highlighting sometimes does not appear when mouse is pressed over handle while speech of element starts
 
 ### External bugs
@@ -31,6 +33,7 @@
 ## Features and enhancements
 
 ### CLI
+
 * Show names of files written to disk. This is useful for cases where a file is auto-renamed to prevent overwriting existing data
 * Restrict input media file extensions to ensure that invalid files are not passed to FFmpeg
 * Show a message when a new version is available
@@ -46,28 +49,35 @@
 * Mode to print IPA words when speaking
 
 ### CLI / playback
+
 * Option to set audio output device for playback
 * Option to set playback volume
 * Maybe find a way not to pre-normalize if the audio is silent (to prevent a 30dB increase of possible noise)
 * Add phone playback support
 
 ### CLI / `speak`
+
 * Add support for sentence templates, like `echogarden speak-file text.txt /parts/[sentence].wav`
 
 ### CLI / `speak-wikipedia`
+
 * Correctly detect language when a Wikipedia URL is passed instead of an article name
 * Add option to set language edition separately from language, since Wikipedia language editions has its own code system that is slightly different from the standard one, in some cases
 
 ### CLI / `speak-url`
+
 * Use the Wikipedia reader when the URL is detected to be from `wikipedia.org`
 
 ### CLI / `list-voices`
+
 * When given a configuration file, see if you can fall back to take options from `speak` options, for example, to take API keys that are required for both the synthesis request and voice list request and
 
 ### CLI / `list-packages`
+
 * Support filters
 
 ### CLI / New commands
+
 * `play-with-subtitles`: Preview subtitles in terminal
 * `play-with-timeline`: Preview timeline in terminal
 * `subtitles-to-text`, `subtitles-to-timeline`, `srt-to-vtt`, `vtt-to-srt`
@@ -78,21 +88,25 @@
 * `speak-youtube-subtitles`: To speak the subtitles of a YouTube video
 
 ### API
+
 * Allow callers from API to cancel a task via `AbortController` and `AbortSignal`
 * Validate timelines to ensure timestamps are always increasing: no undefined timestamps, no negative timestamps, out-of-order timestamps, or timestamps over the duration of the audio. No sentences without words, etc. Missing or incorrect word offsets, etc.
 * Add support for phrases in timelines
 * Accept voice list caching options in `SynthesisOptions`
 
 ### Package manager
+
 * Better error message when a package is not found remotely. Currently, it just gives a `404 not found` without any other information
 * Retry on network failure
 
 ### Speech language detection
 
 ### Text language detection
+
 * Deploy and add the new n-gram based text language detection model
 
 ### Subtitles
+
 * Split long words if needed
 * Decide how many punctuation characters to allow before breaking to a new line (currently it's infinite)
 * If a subtitle is too short and at the end of the audio, try to extend it back if possible (for example, if the previous subtitle is already extended, take back from it)
@@ -101,6 +115,7 @@
 * Parse VTT's language
 
 ### Synthesis
+
 * Option to disable alignment (only for some engines). Alternative: use a low granularity DTW setting that is very fast to compute
 * Find places to add commas (",") to improve speech fluency. VITS voices don't normally add speech breaks if there is no punctuation
 * An isolated dash " - " can maybe be converted to a " , " to ensure there's a break in the speech
@@ -117,6 +132,7 @@
 * When `splitToSentences` is set to `false`, the timeline doesn't include proper sentences. Find a way to pass larger sections to the TTS, but still have proper sentences in the timeline
 
 ### Synthesis / preprocessing
+
 * Full date normalization (e.g. `21 August 2023`, `21 Aug 2023`, `August 21, 2023`)
 * Add support for capitalized-only rules, and possibly also all uppercase / all lowercase rules
 * Add support for multiple consecutive words in `precededBy` and `followedBy` conditions
@@ -130,6 +146,7 @@
 * `≈` symbol as a word character
 
 ### Synthesis / VITS
+
 * Consider adding `⦁︎` (when surrounded by whitespace) as phrase separator
 * Allow limiting how many models are cached in memory.
 * Ensure that caching behaves correctly when the same model is used with different execution providers
@@ -138,49 +155,60 @@
 * Add speaker names to voice list somehow
 
 ### Synthesis / Kokoro
+
 * Consider adding `⦁︎` (when surrounded by whitespace) as phrase separator
 * Ensure that caching behaves correctly when the same model is used with different execution providers
 
 ### Synthesis / Azure Cognitive Services
+
 * Currently, when input is set to be SSML, it is wrapped in a `<speak>` tag. Handle the case where the user made their own SSML document wrapped with a `<speak>` tag as well. Currently, it may send invalid input to Azure
 
 ### Recognition
+
 * Add token UTF-8 bytes in token entries on timeline. Some tokens are only a part of a codepoint, so this will allow to know exactly what they contain - the string representation is would be a Unicode error symbol in that case
 * Show alternatives when playing in the CLI. Clear current line and rewrite already printed text for alternatives during the speech recognition process
 
 ### Recognition / Whisper
+
 * Whisper's Chinese and Japanese output can be split into words in a more accurate way. Consider using a dedicated segmentation library to perform the segmentation in character sequences that have no punctuation characters to aid on guessing word boundaries
-* Cache last model (if enough memory is available). Ensure that caching works when switching between  different execution providers
+* Cache last model (if enough memory is available). Ensure that caching works when switching between different execution providers
 * Whisper timestamp tokens can be used to split into segments, otherwise it is possible to try to guess using pause lengths or voice activity detection
 * Bring back the option to use eSpeak DTW based alignment on segments, as an alternative approach
 
 ### Alignment
 
 ### Alignment / DTW
+
 * For the `granularity` option, add more granularities like `xxx-low` and `xxxx-low` (should the naming be changed? Maybe transition to a new naming scheme?)
 * Add and test official support for more than 6 hours of audio
 
 ### Alignment / DTW-RA
 
 ### Alignment / Whisper
+
 * Show same token stats as recognition now does on `trace` mode
 
 ### Source separation / MDX-NET
+
 * Option to customize overlap
 
 ### Server
+
 * Option to allow or disallow local file paths as arguments to API methods (as a security safeguard)
 
 ### Worker
+
 * Add cancellation checks in more operations
 * Support more operations
 
 ### Browser extension
+
 * Options UI
 * Add supported engines and voices to WebSpeech voice list
 * Pause and resume support
 
 ### Browser extension / content script
+
 * Autoscroll should work even if the scrollbar relevant to the target element is not the viewport's scrollbar
 * Find a way to show handles even for elements that start with a link
 * Add detection for line breaks in `pre` blocks
@@ -216,6 +244,7 @@
 ## Future features and enhancements
 
 ### CLI
+
 * Auto-generate options file, with comments, based on default options of the API
 * Have the CLI launch a background worker (in a thread) to enable better parallelism
 * Playback result audio while synthesis or recognition is still processing in the background
@@ -225,23 +254,29 @@
 * Markdown file as text input?
 
 ### OpenAI compatible local server
+
 * `echogarden serve` would serve a OpenAI-compatible server for all speech recognition, speech translation and speech synthesis engines
 
 ### Web
+
 * Web based frontend UI to the server
 * Adapt some WASM modules to also run on the web
 * Investigate running in WebContainer
 
 ### API
+
 * Auto-install npm modules when needed using an approach similar to like `npm-programmatic`
 
 ### Text enhancement
+
 * Add capitalization and punctuation to recognized outputs if needed (Silero has a model for it for `en`, `de`, `ru`, `es`, but in `.pt` format only)
 
 ### Synthesis
+
 * Synthesize the given subtitle file and try to preserve the existing timing of cues, or even align to existing speech
 
 ### Recognition
+
 * Low latency, streaming recognition mode. Make the partial transcription available as fast as possible
 * Live input / microphone recognition
 * Implement beam search for Whisper decoder
@@ -249,6 +284,7 @@
 * Investigate exporting Whisper models to 16-bit quantized ONNX or a mix of 16-bit and 32-bit
 
 ### Alignment
+
 * Method to align audio file to audio file
 * Allow `dtw` mode work with more speech synthesizers to produce its reference
 * Predict timing for individual letters (graphemes) based on phoneme timestamps (especially useful for Chinese and Japanese)

package/docs/Technical.md

@@ -18,6 +18,7 @@ Packages are downloaded as `.tar.gz` files, and are extracted to `[data-folder]/
 * `/home/user/.local/share/echogarden` on Linux
 
 `[package-id-string]` is formatted as:
+
 ```
 [engine name]-[package id]-[date as yyyymmdd]
 ```

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "echogarden",
-	"version": "3.3.0",
+	"version": "3.4.0",
 	"description": "An easy-to-use speech toolset. Includes tools for synthesis, recognition, alignment, speech translation, language detection, source separation and more.",
 	"author": "Rotem Dan",
 	"license": "MIT AND GPL-3.0",
@@ -44,7 +44,8 @@
 		"tsconfig.json"
 	],
 	"scripts": {
-		"test": "node --experimental-wasi-unstable-preview1 --no-warnings --trace-uncaught ./dist/tests/Test.js",
+		"test": "vitest",
+		"test-manual": "node --experimental-wasi-unstable-preview1 --no-warnings --trace-uncaught ./dist/tests/Test.js",
 		"generate-options-schema": "npx ts-json-schema-generator --path ./src/api/APIOptions.ts --unstable --type APIOptions --tsconfig ./tsconfig.json --out ./data/schemas/options.json",
 		"make-tarballs": "node --no-warnings ./dist/build-tools/MakeTarballsForInstalledPackages.js"
 	},
@@ -52,8 +53,8 @@
 		"echogarden": "./dist/cli/CLILauncher.js"
 	},
 	"dependencies": {
-		"@aws-sdk/client-polly": "~3.1121.0",
-		"@aws-sdk/client-transcribe-streaming": "~3.1121.0",
+		"@aws-sdk/client-polly": "~3.1127.0",
+		"@aws-sdk/client-transcribe-streaming": "~3.1127.0",
 		"@echogarden/audio-io": "~0.4.2",
 		"@echogarden/fasttext-wasm": "~0.1.0",
 		"@echogarden/flite-wasi": "~0.1.1",
@@ -64,7 +65,7 @@
 		"@echogarden/sonic-wasm": "~0.2.0",
 		"@echogarden/speex-resampler-wasm": "~0.3.0",
 		"@echogarden/svoxpico-wasm": "~0.2.0",
-		"@echogarden/text-segmentation": "~0.8.0",
+		"@echogarden/text-segmentation": "~0.8.1",
 		"@echogarden/transformers-nodejs-lite": "~2.17.1-lite.4",
 		"@echogarden/wave-codec": "~0.4.0",
 		"@echogarden/whisper.cpp-binding": "~0.2.2",
@@ -80,11 +81,11 @@
 		"jsdom": "~30.0.1",
 		"kuromoji": "~0.1.2",
 		"microsoft-cognitiveservices-speech-sdk": "~1.51.0",
-		"msgpack-lite": "~0.2.1",
+		"msgpack-lite": "~0.2.2",
 		"onnxruntime-node": "~1.21.1",
-		"openai": "~7.8.0",
-		"quick-json5": "~0.4.3",
-		"regexp-composer": "~0.6.0",
+		"openai": "~7.10.0",
+		"quick-json5": "~0.5.0",
+		"regexp-composer": "~0.7.0",
 		"sam-js": "~0.3.1",
 		"standard-html-escaper": "~0.2.1",
 		"strip-ansi": "~7.2.0",
@@ -92,7 +93,7 @@
 		"tiktoken": "~1.0.22",
 		"tinyld": "~1.3.4",
 		"wasm-feature-detect": "~1.9.0",
-		"wasm-heap-manager": "~0.4.1",
+		"wasm-heap-manager": "~0.5.1",
 		"ws": "~8.21.3",
 		"wtf_wikipedia": "~10.4.2"
 	},
@@ -118,10 +119,11 @@
 		"@types/graceful-fs": "~4.1.9",
 		"@types/jsdom": "~30.0.0",
 		"@types/msgpack-lite": "~0.1.12",
-		"@types/node": "~26.4.0",
+		"@types/node": "~26.5.0",
 		"@types/ws": "~8.18.1",
 		"ts-json-schema-generator": "~2.9.0",
-		"typescript": "~7.0.2"
+		"typescript": "~7.0.2",
+		"vitest": "~5.0.0"
 	},
 	"overrides": {
 		"whatwg-url": "~14.0.0",

package/src/api/Synthesis.ts

@@ -353,7 +353,11 @@ async function synthesizeSegment(text: string, options: SynthesisOptions, callba
 	let language: string
 
 	if (options.language) {
-		language = await normalizeIdentifierToLanguageCode(options.language)
+		try {
+			language = await normalizeIdentifierToLanguageCode(options.language)
+		} catch {
+			language = selectedVoice.languages[0]
+		}
 	} else {
 		language = selectedVoice.languages[0]
 	}
@@ -1814,24 +1818,7 @@ export async function requestVoiceList(options: VoiceListRequestOptions, callbac
 		voiceList = await loadVoiceList()
 	}
 
-	const languageCode = await normalizeIdentifierToLanguageCode(options.language || '')
-
-	if (languageCode) {
-		let filteredVoiceList = voiceList.filter(voice => voice.languages.includes(languageCode))
-
-		if (filteredVoiceList.length == 0 && languageCode.includes('-')) {
-			const shortLanguageCode = getShortLanguageCode(languageCode)
-
-			filteredVoiceList = voiceList.filter(voice => voice.languages.includes(shortLanguageCode))
-		}
-
-		voiceList = filteredVoiceList
-	}
-
-	if (options.voiceGender) {
-		const genderLowercase = options.voiceGender.toLowerCase()
-		voiceList = voiceList.filter(voice => voice.gender == genderLowercase || voice.gender == 'unknown')
-	}
+	let bestMatchingVoice: SynthesisVoice
 
 	if (options.voice) {
 		const namePatternLowerCase = options.voice.toLocaleLowerCase()
@@ -1853,17 +1840,42 @@ export async function requestVoiceList(options: VoiceListRequestOptions, callbac
 				return false
 			})
 		}
-	}
 
-	let bestMatchingVoice = voiceList[0]
+		bestMatchingVoice = voiceList[0]
+	} else {
+		let languageCode = ''
 
-	if (bestMatchingVoice && voiceList.length > 1 && defaultDialectForLanguageCode[languageCode]) {
-		const expandedLanguageCode = defaultDialectForLanguageCode[languageCode]
+		if (options.language) {
+			languageCode = await normalizeIdentifierToLanguageCode(options.language || '')
 
-		for (const voice of voiceList) {
-			if (voice.languages.includes(expandedLanguageCode)) {
-				bestMatchingVoice = voice
-				break
+			let filteredVoiceList = voiceList.filter(voice => voice.languages.includes(languageCode))
+
+			if (filteredVoiceList.length == 0 && languageCode.includes('-')) {
+				const shortLanguageCode = getShortLanguageCode(languageCode)
+
+				filteredVoiceList = voiceList.filter(voice => voice.languages.includes(shortLanguageCode))
+			}
+
+			voiceList = filteredVoiceList
+		}
+
+		if (options.voiceGender) {
+			const genderLowercase = options.voiceGender.toLowerCase()
+
+			voiceList = voiceList.filter(voice => voice.gender == genderLowercase || voice.gender == 'unknown')
+		}
+
+		bestMatchingVoice = voiceList[0]
+
+		if (bestMatchingVoice && voiceList.length > 1 && defaultDialectForLanguageCode[languageCode]) {
+			const expandedLanguageCode = defaultDialectForLanguageCode[languageCode]
+
+			for (const voice of voiceList) {
+				if (voice.languages.includes(expandedLanguageCode)) {
+					bestMatchingVoice = voice
+
+					break
+				}
 			}
 		}
 	}