whispermix 1.4.14 → 1.5.2

package/README.md

@@ -1,6 +1,6 @@
 # 🎙️ WhisperMix
 
-WhisperMix is a flexible module that provides an interface for transcribing audio using OpenAI's Whisper model, Groq's Whisper Large v3 model, or local Whisper models.
+WhisperMix is a flexible module that provides an interface for transcribing audio using OpenAI's Whisper model, Groq's Whisper Large v3 model, local Whisper models, or local Parakeet TDT v3 models.
 
 ## 📦 Installation
 
@@ -36,6 +36,12 @@ const whisperGroq = new WhisperMix({ model: 'groq/whisper-large-v3' }); // For G
 const whisperLocal = new WhisperMix({ model: 'xenova/whisper-large-v3' }); // For local Whisper (large)
 // or
 const whisperLocalBase = new WhisperMix({ model: 'xenova/whisper-base' }); // For local Whisper (base)
+// or
+const whisperParakeet = new WhisperMix({ model: 'istupakov/parakeet-tdt-0.6b-v3' }); // For local Parakeet v3 (int8)
+// or
+const whisperParakeetInt4 = new WhisperMix({ model: 'efederici/parakeet-tdt-0.6b-v3-int4' }); // For local Parakeet v3 (int4 encoder)
+// or
+const whisperParakeetAlt = new WhisperMix({ model: 'nasedkinpv/parakeet-tdt-0.6b-v3-int8' }); // For local Parakeet v3 (alt int8 repo)
 ```
 
 ### 📄 Transcribing from a File
@@ -67,7 +73,24 @@ whisperGroq.fromStream(audioStream)
   .catch(error => console.error(error));
 ```
 
-**Note:** Stream transcription is only available for API-based models (OpenAI and Groq). Local Whisper models require file input.
+**Note:** Stream transcription is only available for API-based models (OpenAI and Groq). Local models (Whisper and Parakeet) require file input.
+
+### 🐦 Parakeet local models
+
+Parakeet TDT v3 local models are downloaded once and cached in:
+
+`~/.cache/whispermix/parakeet/<modelKey>/`
+
+Available local Parakeet model keys:
+
+- `istupakov/parakeet-tdt-0.6b-v3` (about 670 MB, int8)
+- `efederici/parakeet-tdt-0.6b-v3-int4` (about 410 MB, int4/int8 hybrid)
+- `nasedkinpv/parakeet-tdt-0.6b-v3-int8` (about 890 MB, int8)
+
+Notes:
+
+- Parakeet TDT v3 is multilingual (25 European languages).
+- `language` is ignored for Parakeet local models.
 
 ### ⏱️ Long Audio Processing
 
@@ -108,7 +131,7 @@ You can adjust these settings based on your specific rate limiting needs. Note t
 
 Creates a new WhisperMix instance.
 
-- `options.model`: The model to use for transcription. Can be `'openai/whisper-1'` (OpenAI), `'groq/whisper-large-v3'` (Groq), `'xenova/whisper-large-v3'` or `'xenova/whisper-base'` (local).
+- `options.model`: The model to use for transcription. Can be `'openai/whisper-1'` (OpenAI), `'groq/whisper-large-v3'` (Groq), `'xenova/whisper-large-v3'` or `'xenova/whisper-base'` (local Whisper), `'istupakov/parakeet-tdt-0.6b-v3'`, `'efederici/parakeet-tdt-0.6b-v3-int4'`, or `'nasedkinpv/parakeet-tdt-0.6b-v3-int8'` (local Parakeet).
 - `options.bottleneck`: (Optional) Configuration for Bottleneck rate limiting (API models only).
 - `options.chunkSize`: (Optional) The size in seconds of the chunks to split the audio into. Default is 890 seconds.
 - `options.language`: (Optional) Language for local Whisper model. Defaults to 'auto' for automatic detection.

package/SKILL.md

@@ -0,0 +1,88 @@
+---
+name: whispermix
+description: Transcribe audio to text using WhisperMix, a Node.js wrapper around OpenAI Whisper, Groq Whisper Large v3, local Whisper (xenova) and local Parakeet TDT v3 models. Use when the user asks to "transcribe audio", "speech to text", "convert audio/voice to text", mentions an audio file (.mp3, .wav, .m4a, .ogg, .flac, .webm) to turn into text, or names any of these models/providers: Whisper, OpenAI Whisper, Groq Whisper, Whisper Large v3, Parakeet, NVIDIA Parakeet, xenova/whisper, onnx-asr. Also use when the user wants to pick between a cloud API and a local on-device transcription model in a Node.js project.
+---
+
+# WhisperMix
+
+Single Node.js entry point for audio transcription. Picks one of four backends behind the same API: OpenAI Whisper, Groq Whisper Large v3, local Whisper (xenova), or local Parakeet TDT v3. Handles long files by chunking and rate-limits API calls automatically.
+
+Do NOT use for TTS, live microphone streaming, or translation. WhisperMix is one-way: a complete audio file/stream → text.
+
+## Model selection (decide first)
+
+| Need | Pick |
+|---|---|
+| Offline, lowest latency | `istupakov/parakeet-tdt-0.6b-v3` |
+| Offline, smallest footprint (~410 MB) | `efederici/parakeet-tdt-0.6b-v3-int4` |
+| Highest accuracy, cloud | `groq/whisper-large-v3` (fast) or `openai/whisper-1` |
+| Offline + per-language control | `xenova/whisper-large-v3` or `xenova/whisper-base` |
+| Node stream input (not a file) | API only: `openai/whisper-1` or `groq/whisper-large-v3` |
+
+Constraints to surface before coding:
+- Local models (`xenova/*`, `*/parakeet-tdt-0.6b-v3*`) accept **files only**, not streams.
+- `language` option applies only to local Whisper. Parakeet is multilingual and ignores it.
+- API models need `OPENAI_API_KEY` or `GROQ_API_KEY` in the environment.
+- First Parakeet run downloads weights to `~/.cache/whispermix/parakeet/<modelKey>/` — warn the user.
+- Requires `ffmpeg` on `PATH` for long-audio chunking (>15 min split automatically).
+
+## API
+
+```bash
+npm install whispermix
+```
+
+ESM only. If the consumer is CommonJS, use `const WhisperMix = (await import('whispermix')).default;`.
+
+```javascript
+import WhisperMix from 'whispermix';
+
+const w = new WhisperMix({ model: '<modelKey>' });
+const text = await w.fromFile('path/to/audio.mp3');
+// API models only:
+const text2 = await w.fromStream(fs.createReadStream('path/to/audio.mp3'));
+```
+
+Constructor options:
+- `model` (required) — see selection table.
+- `language` — local Whisper only, e.g. `'spanish'`. Default `'auto'`.
+- `chunkSize` — seconds per chunk for long audio. Default `890` (~14m50s).
+- `bottleneck` — Bottleneck config for API models. Defaults: `minTime: 3000`, `maxConcurrent: 1`, `reservoir: 18`, `reservoirRefreshAmount: 18`, `reservoirRefreshInterval: 60000`.
+- `showProgress` — boolean, prints chunk/decoding progress.
+
+## Examples
+
+Cheapest local:
+
+```javascript
+import WhisperMix from 'whispermix';
+const w = new WhisperMix({ model: 'efederici/parakeet-tdt-0.6b-v3-int4', showProgress: true });
+console.log(await w.fromFile('meeting.wav'));
+```
+
+Groq with custom rate limit:
+
+```javascript
+import WhisperMix from 'whispermix';
+const w = new WhisperMix({
+    model: 'groq/whisper-large-v3',
+    bottleneck: { minTime: 4000, maxConcurrent: 1 },
+});
+console.log(await w.fromFile('podcast.mp3'));
+```
+
+Local Whisper, fixed language:
+
+```javascript
+import WhisperMix from 'whispermix';
+const w = new WhisperMix({ model: 'xenova/whisper-large-v3', language: 'spanish' });
+console.log(await w.fromFile('entrevista.m4a'));
+```
+
+## Troubleshooting
+
+- **`OPENAI_API_KEY`/`GROQ_API_KEY is not set`** — export the key, or switch to a local model.
+- **`fromStream` not supported** — local models are file-only. Use `fromFile`, or switch to an API model.
+- **`Cannot find ffmpeg`** — install it (`brew install ffmpeg` / `apt install ffmpeg`).
+- **First Parakeet call hangs** — weights downloading; enable `showProgress: true`.
+- **`ERR_REQUIRE_ESM`** — WhisperMix is ESM-only; use dynamic `import()` from CommonJS.

package/demo/parakeet.js

@@ -0,0 +1,9 @@
+import WhisperMix from '../index.js';
+
+const whisperParakeet = new WhisperMix({
+    model: 'efederici/parakeet-tdt-0.6b-v3-int4',
+    showProgress: true,
+});
+
+const result = await whisperParakeet.fromFile('conversation.wav');
+console.log(result);

package/index.js

@@ -4,11 +4,52 @@ import path from 'path';
 import ffmpeg from 'fluent-ffmpeg';
 import { getAudioDurationInSeconds } from 'get-audio-duration';
 import os from 'os'; // For temporary directory
+import { Readable } from 'stream';
+import { pipeline as streamPipeline } from 'stream/promises';
 
 // Static imports for local dependencies
-import { pipeline, env } from '@huggingface/transformers';
+import { pipeline as hfPipeline, env } from '@huggingface/transformers';
 import audioDecode from 'audio-decode';
 
+const PARAKEET_LAYOUTS = {
+    'istupakov/parakeet-tdt-0.6b-v3': {
+        cacheKey: 'istupakov-parakeet-tdt-0.6b-v3',
+        repo: 'istupakov/parakeet-tdt-0.6b-v3-onnx',
+        files: [
+            { src: 'config.json', dst: 'config.json' },
+            { src: 'nemo128.onnx', dst: 'nemo128.onnx' },
+            { src: 'encoder-model.int8.onnx', dst: 'encoder-model.int8.onnx' },
+            { src: 'decoder_joint-model.int8.onnx', dst: 'decoder_joint-model.int8.onnx' },
+            { src: 'vocab.txt', dst: 'vocab.txt' },
+        ],
+    },
+    'efederici/parakeet-tdt-0.6b-v3-int4': {
+        cacheKey: 'efederici-parakeet-tdt-0.6b-v3-int4',
+        repo: 'efederici/parakeet-tdt-0.6b-v3-onnx-int4',
+        files: [
+            { src: 'config.json', dst: 'config.json' },
+            { src: 'nemo128.onnx', dst: 'nemo128.onnx' },
+            { src: 'encoder-model.int4.onnx', dst: 'encoder-model.int8.onnx' },
+            { src: 'decoder_joint-model.int8.onnx', dst: 'decoder_joint-model.int8.onnx' },
+            { src: 'vocab.txt', dst: 'vocab.txt' },
+        ],
+    },
+    'nasedkinpv/parakeet-tdt-0.6b-v3-int8': {
+        cacheKey: 'nasedkinpv-parakeet-tdt-0.6b-v3-int8',
+        repo: 'nasedkinpv/parakeet-tdt-0.6b-v3-onnx-int8',
+        files: [
+            { src: 'encoder-int8.onnx', dst: 'encoder-model.int8.onnx' },
+            { src: 'encoder-int8.onnx.data', dst: 'encoder-model.int8.onnx.data' },
+            { src: 'decoder_joint-int8.onnx', dst: 'decoder_joint-model.int8.onnx' },
+            { src: 'vocab.txt', dst: 'vocab.txt' },
+            { srcRepo: 'istupakov/parakeet-tdt-0.6b-v3-onnx', src: 'nemo128.onnx', dst: 'nemo128.onnx' },
+        ],
+        synthConfig: {
+            model_type: 'nemo-conformer-tdt',
+        },
+    },
+};
+
 class WhisperMix {
     constructor(setup = {}) {
         this.model = 'openai/whisper-1';
@@ -41,7 +82,26 @@ class WhisperMix {
                 local: true,
                 modelName: 'Xenova/whisper-base',
                 dtype: 'q8',
-            },            
+                backend: 'transformers',
+            },
+            'istupakov/parakeet-tdt-0.6b-v3': {
+                local: true,
+                modelName: 'Parakeet-TDT-0.6B-v3',
+                backend: 'onnx-asr-web',
+                layout: PARAKEET_LAYOUTS['istupakov/parakeet-tdt-0.6b-v3'],
+            },
+            'efederici/parakeet-tdt-0.6b-v3-int4': {
+                local: true,
+                modelName: 'Parakeet-TDT-0.6B-v3-int4',
+                backend: 'onnx-asr-web',
+                layout: PARAKEET_LAYOUTS['efederici/parakeet-tdt-0.6b-v3-int4'],
+            },
+            'nasedkinpv/parakeet-tdt-0.6b-v3-int8': {
+                local: true,
+                modelName: 'Parakeet-TDT-0.6B-v3-int8',
+                backend: 'onnx-asr-web',
+                layout: PARAKEET_LAYOUTS['nasedkinpv/parakeet-tdt-0.6b-v3-int8'],
+            },
         };
 
         Object.assign(this, setup);
@@ -56,7 +116,11 @@ class WhisperMix {
         this.modelName = this.config.modelName;
         this.dtype = this.dtype || this.config.dtype;
         this.showProgress = this.showProgress || false;
+        this.localBackend = this.config.backend || 'transformers';
+        this.layout = this.config.layout;
         this.transcriber = null;
+        this._onnxAsrNodeModule = null;
+        this._warnedParakeetLanguage = false;
 
         this.limiter = new Bottleneck(this.bottleneck);
     }
@@ -212,6 +276,19 @@ class WhisperMix {
             // Reuse a single pipeline instance so repeated calls don't re-download/re-initialize.
             const transcriber = await this._getLocalTranscriber();
 
+            if (this.localBackend === 'onnx-asr-web') {
+                if (this.language !== undefined && this.showProgress && !this._warnedParakeetLanguage) {
+                    console.log('[WhisperMix] "language" option is ignored for Parakeet local models.');
+                    this._warnedParakeetLanguage = true;
+                }
+                const result = await transcriber.transcribeSamples(audioData, 16000);
+                const text = result?.text || result?.utterance_text;
+                if (typeof text !== 'string') {
+                    throw new Error('Parakeet transcription returned no text output.');
+                }
+                return text.trim();
+            }
+
             // Pass the processed audio data
             const transcriberOptions = {
                 task: 'transcribe',
@@ -220,15 +297,32 @@ class WhisperMix {
                 transcriberOptions.language = this.language;
             }
             const result = await transcriber(audioData, transcriberOptions);
-
             return result.text.trim();
         } catch (error) {
-            throw new Error(`Local transcription failed: ${error.message}. If this happened after an interrupted download, remove the model cache at ${env.cacheDir}${this.modelName}/ and try again.`);
+            const cacheHint = this.localBackend === 'onnx-asr-web'
+                ? this._getParakeetCacheDir(this.layout)
+                : `${env.cacheDir}${this.modelName}/`;
+            throw new Error(`Local transcription failed: ${error.message}. If this happened after an interrupted download, remove the model cache at ${cacheHint} and try again.`);
         }
     }
 
     async _getLocalTranscriber() {
         if (!this.transcriber) {
+            if (this.localBackend === 'onnx-asr-web') {
+                if (!this.layout) {
+                    throw new Error(`Missing Parakeet layout for model: ${this.model}`);
+                }
+                const modelDir = await this._ensureParakeetAssets(this.layout);
+                const { loadLocalModel } = await this._getOnnxAsrNodeModule();
+                this.transcriber = await loadLocalModel(modelDir, {
+                    quantization: 'int8',
+                    sessionOptions: {
+                        executionProviders: ['wasm'],
+                    },
+                });
+                return this.transcriber;
+            }
+
             const options = {};
             if (this.dtype) {
                 options.dtype = this.dtype;
@@ -244,11 +338,90 @@ class WhisperMix {
                 };
             }
 
-            this.transcriber = pipeline('automatic-speech-recognition', this.modelName, options);
+            this.transcriber = hfPipeline('automatic-speech-recognition', this.modelName, options);
         }
 
         return this.transcriber;
     }
+
+    async _getOnnxAsrNodeModule() {
+        if (!this._onnxAsrNodeModule) {
+            this._onnxAsrNodeModule = await import('onnx-asr-web/node');
+        }
+        return this._onnxAsrNodeModule;
+    }
+
+    _getParakeetCacheDir(layout) {
+        if (this.cacheDir) {
+            return path.resolve(this.cacheDir);
+        }
+        if (!layout?.cacheKey) {
+            throw new Error(`Missing cache key for Parakeet layout: ${this.model}`);
+        }
+        return path.join(os.homedir(), '.cache', 'whispermix', 'parakeet', layout.cacheKey);
+    }
+
+    async _ensureParakeetAssets(layout) {
+        const cacheDir = this._getParakeetCacheDir(layout);
+        await fs.promises.mkdir(cacheDir, { recursive: true });
+
+        for (const fileDef of layout.files) {
+            const destination = path.join(cacheDir, fileDef.dst);
+            const sourceRepo = fileDef.srcRepo || layout.repo;
+
+            if (await this._hasFileWithContent(destination)) {
+                continue;
+            }
+
+            const sourceUrl = `https://huggingface.co/${sourceRepo}/resolve/main/${fileDef.src}`;
+            if (this.showProgress) {
+                console.log(`[WhisperMix] Downloading ${fileDef.src} from ${sourceRepo}`);
+            }
+            await this._downloadToFile(sourceUrl, destination);
+        }
+
+        if (layout.synthConfig) {
+            const configPath = path.join(cacheDir, 'config.json');
+            if (!(await this._hasFileWithContent(configPath))) {
+                await fs.promises.writeFile(configPath, JSON.stringify(layout.synthConfig, null, 2));
+            }
+        }
+
+        return cacheDir;
+    }
+
+    async _hasFileWithContent(filePath) {
+        try {
+            const stats = await fs.promises.stat(filePath);
+            return stats.isFile() && stats.size > 0;
+        } catch {
+            return false;
+        }
+    }
+
+    async _downloadToFile(url, filePath) {
+        const response = await fetch(url);
+        if (!response.ok) {
+            throw new Error(`Download failed (${response.status} ${response.statusText}) for ${url}`);
+        }
+        if (!response.body) {
+            throw new Error(`Empty response body while downloading ${url}`);
+        }
+
+        await fs.promises.mkdir(path.dirname(filePath), { recursive: true });
+        const tempPath = `${filePath}.tmp-${process.pid}-${Date.now()}`;
+
+        try {
+            await streamPipeline(
+                Readable.fromWeb(response.body),
+                fs.createWriteStream(tempPath),
+            );
+            await fs.promises.rename(tempPath, filePath);
+        } catch (error) {
+            await fs.promises.rm(tempPath, { force: true });
+            throw error;
+        }
+    }
 }
 
 // Support both CommonJS and ES modules

package/package.json

@@ -1,57 +1,62 @@
 {
-  "name": "whispermix",
-  "description": "🎙️ WhisperMix is a versatile module for transcribing audio using OpenAI’s Whisper or Groq’s Whisper v3 model.",
-  "version": "1.4.14",
-  "type": "module",
-  "keywords": [
-    "whisper",
-    "openai",
-    "groq",
-    "transcription",
-    "speech-to-text",
-    "audio",
-    "voice",
-    "ai",
-    "machine-learning",
-    "nlp",
-    "natural-language-processing",
-    "audio-processing",
-    "voice-recognition",
-    "speech-recognition",
-    "api-wrapper",
-    "whisper-large-v3",
-    "whisper-1",
-    "streaming",
-    "file-processing",
-    "multilingual",
-    "bottleneck",
-    "chunk",
-    "clasen",
-    "local",
-    "xenova"
-  ],
-  "repository": {
-    "type": "git",
-    "url": "git+https://github.com/clasen/WhisperMix.git"
-  },
-  "main": "index.js",
-  "author": "Martin Clasen",
-  "license": "MIT",
-  "bugs": {
-    "url": "https://github.com/clasen/WhisperMix/issues"
-  },
-  "dependencies": {
-    "@huggingface/transformers": "^4.2.0",
-    "audio-decode": "^2.2.3",
-    "bottleneck": "^2.19.5",
-    "fluent-ffmpeg": "^2.1.3",
-    "get-audio-duration": "^4.0.1"
-  },
-  "devDependencies": {
-    "node-addon-api": "^8.7.0",
-    "node-gyp": "^12.3.0"
-  },
-  "scripts": {
-    "test": "echo \"Error: no test specified\" && exit 1"
-  }
-}
+    "name": "whispermix",
+    "description": "🎙️ WhisperMix is a versatile module for transcribing audio using OpenAI’s Whisper or Groq’s Whisper v3 model.",
+    "version": "1.5.2",
+    "type": "module",
+    "keywords": [
+        "whisper",
+        "openai",
+        "groq",
+        "transcription",
+        "speech-to-text",
+        "audio",
+        "voice",
+        "ai",
+        "machine-learning",
+        "nlp",
+        "natural-language-processing",
+        "audio-processing",
+        "voice-recognition",
+        "speech-recognition",
+        "api-wrapper",
+        "whisper-large-v3",
+        "whisper-1",
+        "streaming",
+        "file-processing",
+        "multilingual",
+        "bottleneck",
+        "chunk",
+        "clasen",
+        "local",
+        "xenova",
+        "parakeet",
+        "nvidia",
+        "onnx"
+    ],
+    "repository": {
+        "type": "git",
+        "url": "git+https://github.com/clasen/WhisperMix.git"
+    },
+    "main": "index.js",
+    "scripts": {
+        "test": "echo \"Error: no test specified\" && exit 1"
+    },
+    "author": "Martin Clasen",
+    "license": "MIT",
+    "bugs": {
+        "url": "https://github.com/clasen/WhisperMix/issues"
+    },
+    "dependencies": {
+        "@huggingface/transformers": "^4.2.0",
+        "audio-decode": "^2.2.3",
+        "bottleneck": "^2.19.5",
+        "fluent-ffmpeg": "^2.1.3",
+        "get-audio-duration": "^4.0.1",
+        "onnx-asr-web": "^0.1.3"
+    },
+    "devDependencies": {
+        "node-addon-api": "^8.7.0",
+        "node-gyp": "^12.3.0"
+    },
+    "packageManager": "pnpm@11.1.2"
+}