parakeet.js 0.0.1

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Yunus Seyhan Dede
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE. 

package/README.md

@@ -0,0 +1,210 @@
+# Parakeet.js  
+
+Client-side ONNX inference of NVIDIA *Parakeet* speech-to-text models.
+Runs entirely in the browser on **WebGPU** or **WASM** via
+[ONNX Runtime Web](https://onnxruntime.ai/).
+
+> **Parakeet.js** offers a high-performance, browser-first implementation for NVIDIA's Parakeet-TDT speech-to-text models, running entirely client-side via WebGPU and WASM. Powered by ONNX Runtime Web, this library makes it simple to integrate state-of-the-art transcription into any web application.
+
+> **Status:** Early preview – API is subject to change while things stabilise.
+> **Note:** Currently only supports the Parakeet-TDT model architecture.
+
+---
+
+## Installation
+
+```bash
+# npm
+npm i parakeet.js onnxruntime-web
+
+# yarn
+yarn add parakeet.js onnxruntime-web
+```
+
+`onnxruntime-web` is a peer-dependency that supplies the runtime back-ends (WebGPU, WASM).
+
+---
+
+## Model assets
+
+We host ready-to-use ONNX exports on the HuggingFace Hub:
+
+```
+ysdede/parakeet-tdt-0.6b-v2-onnx
+```
+
+The helper `getParakeetModel()` downloads all required files and caches them in **IndexedDB**:
+
+```js
+import { getParakeetModel } from 'parakeet.js';
+
+const repoId = 'ysdede/parakeet-tdt-0.6b-v2-onnx';
+const { urls, filenames } = await getParakeetModel(repoId, {
+  backend: 'webgpu-hybrid', // webgpu-hybrid | wasm
+  quantization: 'fp32',     // fp32 | int8
+  decoderInt8: true,        // load INT8 decoder even when encoder fp32
+  preprocessor: 'nemo128',  // nemo80 | nemo128
+  progress: ({file,loaded,total}) => console.log(file, loaded/total)
+});
+```
+
+Returned structure:
+
+```ts
+{
+  urls: {
+    encoderUrl: string,
+    decoderUrl: string,
+    encoderDataUrl?: string | null,
+    decoderDataUrl?: string | null,
+    tokenizerUrl: string,
+    preprocessorUrl: string
+  },
+  filenames: { encoder: string; decoder: string },
+  quantisation: { encoder: 'fp32' | 'int8'; decoder: 'fp32' | 'int8' }
+}
+```
+
+---
+
+## Creating a model instance
+
+```js
+import { ParakeetModel } from 'parakeet.js';
+
+const model = await ParakeetModel.fromUrls({
+  ...urls,                 // spread the URLs returned above
+  filenames,               // needed for external .data mapping
+  backend: 'webgpu-hybrid',
+  decoderOnWasm: true,     // force decoder to CPU/WASM for micro-kernels
+  decoderInt8: true,       // decoder uses INT8 weights
+  cpuThreads: 6,           // WASM threads (defaults to cores-2)
+  verbose: false           // ORT verbose log
+});
+```
+
+### Back-end presets
+
+| Backend string      | Encoder EP | Decoder EP | Typical use-case |
+|---------------------|------------|------------|------------------|
+| `webgpu-hybrid` (default) | WebGPU (fp32) | WASM (fp32/int8) | Modern desktop browsers |
+| `webgpu-strict`    | WebGPU (fp32) | **fail** if op unsupported | Benchmarking kernels |
+| `wasm`             | WASM (int8/fp32) | WASM | Low-end devices, Node.js |
+
+---
+
+## Transcribing audio
+
+```js
+// 16-kHz mono PCM Float32Array
+await model.transcribe(pcmFloat32, 16_000, {
+  returnTimestamps: true,
+  returnConfidences: true,
+  frameStride: 2,      // 1 (default) = highest accuracy / 2-4 faster
+});
+```
+
+Extra options:
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `temperature` | 1.2 | Softmax temperature for decoding |
+| `frameStride` | 1 | Advance decoder by *n* encoder frames per step |
+
+### Result schema
+
+```ts
+{
+  utterance_text: string,
+  words: Array<{text,start_time,end_time,confidence}>,
+  tokens: Array<{token,start_time,end_time,confidence}>,
+  confidence_scores: { overall_log_prob, word_avg, token_avg },
+  metrics: {
+    rtf: number,
+    total_ms: number,
+    preprocess_ms: number,
+    encode_ms: number,
+    decode_ms: number,
+    tokenize_ms: number
+  },
+  is_final: true
+}
+```
+
+---
+
+## Warm-up & Verification (Recommended)
+
+The first time you run inference after loading a model, the underlying runtime needs to compile the execution graph. This makes the first run significantly slower. To ensure a smooth user experience, it's best practice to perform a "warm-up" run with a dummy or known audio sample immediately after model creation.
+
+Our React demo does this and also verifies the output to ensure the model loaded correctly.
+
+```js
+// In your app, after `ParakeetModel.fromUrls()` succeeds:
+setStatus('Warming up & verifying…');
+
+const audioRes = await fetch('/assets/known_audio.wav');
+const pcm = await decodeAudio(audioRes); // Your audio decoding logic
+const { utterance_text } = await model.transcribe(pcm, 16000);
+
+const expected = 'the known transcript for your audio';
+if (utterance_text.toLowerCase().includes(expected)) {
+  setStatus('Model ready ✔');
+} else {
+  setStatus('Model verification failed!');
+}
+```
+
+---
+
+## Runtime tuning knobs
+
+| Property | Where | Effect |
+|----------|-------|--------|
+| `cpuThreads` | `fromUrls()` | Sets `ort.env.wasm.numThreads`; pick *cores-2* for best balance |
+| `decoderOnWasm` | `fromUrls()` | Forces decoder session to WASM even in hybrid mode |
+| `decoderInt8` | `getParakeetModel()` + `fromUrls()` | Load INT8 weights for decoder only |
+| `frameStride` | `transcribe()` | Trade-off latency vs accuracy |
+| `enableProfiling` | `fromUrls()` | Enables ORT profiler (JSON written to `/tmp/profile_*.json`) |
+
+---
+
+## Using the React demo as a template
+
+Located at `examples/react-demo`.
+
+Quick start:
+
+```bash
+cd examples/react-demo
+npm i
+npm run dev  # Vite => http://localhost:5173
+```
+
+Key components:
+
+| File | Purpose |
+|------|---------|
+| `App.jsx` | Complete end-to-end reference UI. Shows how to load a model with progress bars, perform a warm-up/verification step, display performance metrics (RTF, timings), and manage transcription history. |
+| `parakeet.js` | Library entry; houses the model wrapper and performance instrumentation. |
+| `hub.js` | Lightweight HuggingFace Hub helper – downloads and caches model binaries. |
+
+Copy-paste the `loadModel()` and `transcribeFile()` functions into your app, adjust UI bindings, and you are ready to go.
+
+---
+
+## Troubleshooting
+
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| `multiple calls to initWasm()` | Two WASM sessions initialised in parallel | In hybrid mode we create encoder session first, then decoder. Keep this order. |
+| GPU memory still ~2.4 GB with INT8 selected | WebGPU kernels don't support INT8 yet – weights are automatically converted to FP32 | Use `decoderInt8:true` (CPU) or wait for upcoming WebGPU INT8 kernels. |
+| `Graph capture feature not available` error | Mixed EPs prevent GPU graph capture | We auto-retry without capture; nothing to do. |
+
+---
+
+## Changelog
+
+See `OPTIMIZATION_PLAN.md` for a timeline of performance tweaks and planned features.
+
+Happy hacking! 🎉

package/examples/react-demo/index.html

@@ -0,0 +1,12 @@
+<!DOCTYPE html>
+<html lang="en">
+  <head>
+    <meta charset="UTF-8" />
+    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <title>Parakeet React Demo</title>
+  </head>
+  <body>
+    <div id="root"></div>
+    <script type="module" src="/src/main.jsx"></script>
+  </body>
+</html> 

package/examples/react-demo/package.json

@@ -0,0 +1,20 @@
+{
+  "name": "parakeet-react-demo",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "dev": "vite",
+    "build": "vite build",
+    "preview": "vite preview"
+  },
+  "dependencies": {
+    "react": "^18.2.0",
+    "react-dom": "^18.2.0",
+    "parakeet.js": "file:../..",
+    "onnxruntime-web": "1.22.0-dev.20250409-89f8206ba4"
+  },
+  "devDependencies": {
+    "vite": "^4.5.0",
+    "@vitejs/plugin-react": "^4.0.0"
+  }
+} 

package/examples/react-demo/src/App.css

@@ -0,0 +1,134 @@
+:root {
+  font-family: Inter, system-ui, sans-serif;
+  line-height: 1.4;
+  color: #222;
+  background: #f3f6f8;
+}
+
+.app {
+  max-width: 760px;
+  margin: 2rem auto;
+  background: #ffffff;
+  border-radius: 8px;
+  padding: 1.5rem 2rem;
+  box-shadow: 0 4px 14px rgba(0, 0, 0, 0.06);
+}
+
+.controls {
+  display: flex;
+  flex-wrap: wrap;
+  gap: 0.75rem;
+  align-items: center;
+  margin-bottom: 1rem;
+}
+
+.controls label {
+  font-size: 0.9rem;
+  display: flex;
+  align-items: center;
+  gap: 0.35rem;
+}
+
+.controls select,
+.controls input[type="number"] {
+  padding: 0.25rem 0.5rem;
+  border: 1px solid #d1d5db;
+  border-radius: 4px;
+  background: #fff;
+}
+
+button.primary {
+  padding: 0.4rem 0.9rem;
+  background: #3b82f6;
+  color: #ffffff;
+  border: none;
+  border-radius: 4px;
+  cursor: pointer;
+}
+
+button.primary:hover {
+  background: #2563eb;
+}
+
+.status {
+  margin-top: 0.5rem;
+  font-weight: 500;
+}
+
+.progress-wrapper {
+  margin: 0.5rem 0;
+}
+
+.progress-bar {
+  height: 8px;
+  background: #e2e8f0;
+  border-radius: 4px;
+  overflow: hidden;
+}
+
+.progress-bar > div {
+  height: 100%;
+  background: #10b981;
+  transition: width 0.2s;
+}
+
+.progress-text {
+  font-size: 0.8rem;
+  color: #555;
+  margin-top: 0.25rem;
+}
+
+.textarea {
+  width: 100%;
+  height: 6rem;
+  resize: vertical;
+}
+
+.performance {
+  font-size: 0.85rem;
+  background: #ecfdf5;
+  padding: 0.5rem 0.75rem;
+  border-radius: 6px;
+  border: 1px solid #d1fae5;
+  margin-bottom: 1rem;
+}
+
+.history {
+  max-height: 400px;
+  overflow-y: auto;
+  border: 1px solid #e2e8f0;
+  border-radius: 6px;
+  background: #ffffff;
+}
+
+.history-item {
+  padding: 1rem;
+  border-bottom: 1px solid #f1f5f9;
+  background: #ffffff;
+}
+
+.history-item:last-child {
+  border-bottom: none;
+}
+
+.history-meta {
+  display: flex;
+  justify-content: space-between;
+  font-size: 0.9rem;
+  color: #666;
+  margin-bottom: 0.5rem;
+}
+
+.history-stats {
+  font-size: 0.75rem;
+  color: #666;
+  margin-bottom: 0.5rem;
+}
+
+.history-text {
+  background: #f9fafb;
+  padding: 0.5rem 0.75rem;
+  border-radius: 4px;
+  border: 1px solid #e5e7eb;
+  font-size: 0.9rem;
+}