@rivus/gateway 0.18.2 → 0.19.0

package/README.md

@@ -19,6 +19,10 @@ Gateway is the sole owner of deployment templates, the starter Plugin and runtim
 `@rivus/gateway/plugin/starter` and `@rivus/gateway/skills/runtime-management/SKILL.md` directly. Background-session
 MCP uses `@rivus/gateway/mcp`; old Agent bootstrap, Plugin, Skill and MCP aliases are not available.
 
+Gateway also ships the installer assets of the Rivus-managed audio transcriber under `assets/audio-transcriber/`.
+`rivus audio install` downloads the engine, its npm packages and the model onto the host at install time; the package
+contains none of them.
+
 Pi deployment Bootstrap opens the Runtime JSONL Memory Repository and injects one canonical Effect Memory service.
 The former Gateway Memory service factories are removed; use Runtime's public Repository and service constructors.
 

package/assets/audio-transcriber/THIRD_PARTY_NOTICES.md

@@ -0,0 +1,86 @@
+# Third-Party Notices
+
+The Rivus-managed audio transcriber ships only its own scripts. `rivus audio install` downloads the components below
+from their upstream sources into the installed release and verifies them against the pins in `package.json`,
+`npm-shrinkwrap.json` and `manifest.json`. Each component keeps its own license.
+
+## Models
+
+| Component | Source | License |
+| --- | --- | --- |
+| OpenAI Whisper turbo (native runtime) or small (WebAssembly runtime) weights and tokenizer, exported to int8 ONNX by sherpa-onnx | `sherpa-onnx-whisper-turbo.tar.bz2` / `sherpa-onnx-whisper-small.tar.bz2`, `k2-fsa/sherpa-onnx` GitHub release `asr-models` | MIT, Copyright (c) 2022 OpenAI (text below) |
+| Silero VAD (`silero_vad.int8.onnx`) | `k2-fsa/sherpa-onnx` GitHub release `asr-models` | MIT, Copyright (c) 2020-present Silero Team (text below) |
+| Install smoke-test recording (`model/smoke.wav`) | `test_wavs/0.wav`, a test file inside the same sherpa-onnx Whisper archive. Its English transcript appears to be a LibriSpeech utterance, but the archive names no source for it. Rivus does not redistribute it: the installer extracts it on the deployment host and uses it only as a local smoke test. | Not stated in the archive |
+
+The sherpa-onnx model archives contain no license file of their own. The Whisper weights are the ones published by
+OpenAI in <https://github.com/openai/whisper>, whose README states that the code and the model weights are released
+under the MIT License.
+
+## npm packages
+
+| Package | License |
+| --- | --- |
+| `sherpa-onnx-node`, its platform packages (`sherpa-onnx-<os>-<cpu>`) and `sherpa-onnx` (WebAssembly) 1.13.8 | Apache-2.0. The binaries bundle onnxruntime (MIT) and statically include espeak-ng (GPL-3.0), which only the text-to-speech functions use. |
+| `ogg-opus-decoder` 1.7.5, `opus-decoder`, `@wasm-audio-decoders/*`, `simple-yenc` | MIT; the decoders are compiled from libopus (BSD-3-Clause) |
+| `codec-parser` | LGPL-3.0-or-later |
+| `@eshaz/web-worker` | Apache-2.0 |
+| `opencc-js` 1.4.2 | MIT AND Apache-2.0 (code MIT; dictionary data generated from `opencc-data`, Apache-2.0) |
+| `unbzip2-stream` 1.4.3, `through`, `buffer`, `base64-js` | MIT |
+| `ieee754` | BSD-3-Clause |
+| `tar` 7.5.22, `minipass`, `chownr`, `yallist` | BlueOak-1.0.0 |
+| `minizlib` | MIT |
+| `@isaacs/fs-minipass` | ISC |
+
+Each installed package carries its license text under `node_modules/<package>`.
+
+## OpenAI Whisper
+
+```text
+MIT License
+
+Copyright (c) 2022 OpenAI
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+```
+
+## Silero VAD
+
+```text
+MIT License
+
+Copyright (c) 2020-present Silero Team
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+```

package/assets/audio-transcriber/byte-tokens.mjs

@@ -0,0 +1,33 @@
+// Whisper's byte-level BPE splits many CJK characters across tokens, so one token can hold an incomplete UTF-8
+// sequence. sherpa-onnx 1.13.8 drops invalid UTF-8 from every token before joining them, which deletes those characters
+// (fixed upstream after 1.13.8 by k2-fsa/sherpa-onnx#3965). fetch-model.mjs therefore installs tokens.bytes.txt, which
+// spells every non-ASCII token byte as the code point BYTE_BASE + byte, valid UTF-8 on its own; transcribe.mjs maps the
+// recognizer's text back to bytes.
+const BYTE_BASE = 0xe000;
+
+/** Rewrites tokens.txt, whose lines are "<base64 of the token's bytes> <id>", into tokens.bytes.txt. */
+export function spellByteTokens(tokensText) {
+  const lines = tokensText.split("\n").filter(Boolean);
+  return lines
+    .map((line) => {
+      const [base64, id] = line.split(" ");
+      if (!/^\d+$/.test(id ?? "")) {
+        throw new Error(`unexpected tokens.txt line: ${line.slice(0, 40)}`);
+      }
+      const text = [...Buffer.from(base64, "base64")]
+        .map((byte) => String.fromCodePoint(byte < 0x80 ? byte : BYTE_BASE + byte))
+        .join("");
+      // The last entry ("= 50256") decodes to no bytes and stays as it is.
+      return text ? `${Buffer.from(text, "utf8").toString("base64")} ${id}\n` : `${line}\n`;
+    })
+    .join("");
+}
+
+/** Turns recognizer text back into bytes and decodes them. A chunk cut off inside a character drops that character. */
+export function decodeByteText(text) {
+  const bytes = [...text].map((character) => {
+    const code = character.codePointAt(0);
+    return code >= BYTE_BASE + 0x80 && code <= BYTE_BASE + 0xff ? code - BYTE_BASE : code;
+  });
+  return Buffer.from(bytes).toString("utf8").replaceAll("\uFFFD", "");
+}

package/assets/audio-transcriber/fetch-model.mjs

@@ -0,0 +1,225 @@
+// Installs the transcriber model into <model-dir>, all or nothing. Usage: node fetch-model.mjs <manifest.json> <model-dir>
+//
+// The engine runtime is chosen here, once per installation: the native sherpa-onnx addon when it loads on this host
+// (on Linux x64 it needs glibc 2.32 and the libstdc++ of GCC 11), otherwise the WebAssembly build, which runs wherever
+// Node runs but on one thread. Each runtime has its own Whisper size in the manifest, and the choice is recorded in
+// <model-dir>/runtime.json for transcribe.mjs.
+//
+// RIVUS_AUDIO_RUNTIME=auto|native|wasm (default auto) forces a runtime. RIVUS_AUDIO_MODEL_BASE_URL replaces the
+// manifest's baseUrl with a mirror of the sherpa-onnx GitHub releases; the sha256 pins still apply.
+//
+// A mirror is untrusted, so no downloaded byte is parsed before its file matches the manifest's size and sha256: the
+// model archive is downloaded completely into the staging directory and only then unpacked, which needs room for the
+// archive and its extracted members at the same time.
+import { createHash } from "node:crypto";
+import { createReadStream, createWriteStream } from "node:fs";
+import { mkdir, readFile, rename, rm, stat, writeFile } from "node:fs/promises";
+import { createRequire } from "node:module";
+import { join, resolve } from "node:path";
+import { Readable, Transform } from "node:stream";
+import { pipeline } from "node:stream/promises";
+
+import { extract } from "tar";
+import bunzip2 from "unbzip2-stream";
+
+import { spellByteTokens } from "./byte-tokens.mjs";
+
+const RUNTIMES = ["auto", "native", "wasm"];
+// A download that delivers no bytes for this long is aborted and tried once more.
+const STALL_MS = 60_000;
+const ATTEMPTS = 2;
+
+class IntegrityError extends Error {}
+
+async function chooseRuntime(requested) {
+  if (!RUNTIMES.includes(requested)) {
+    throw new Error(`RIVUS_AUDIO_RUNTIME must be one of ${RUNTIMES.join(", ")}, not ${JSON.stringify(requested)}`);
+  }
+  if (requested === "wasm") {
+    return { runtime: "wasm" };
+  }
+  try {
+    await import("sherpa-onnx-node");
+    return { runtime: "native" };
+  } catch (error) {
+    const reason = nativeLoadError(error);
+    if (requested === "native") {
+      throw new Error(`the native sherpa-onnx addon does not load on this host: ${reason}`);
+    }
+    return { nativeError: reason, runtime: "wasm" };
+  }
+}
+
+// sherpa-onnx-node's loader replaces the dlopen error with a list of the paths it tried. Loading the platform binary
+// directly reports the actual cause, such as a missing package or GLIBC version.
+function nativeLoadError(error) {
+  const platform = process.platform === "win32" ? "win" : process.platform;
+  try {
+    createRequire(import.meta.url)(`sherpa-onnx-${platform}-${process.arch}/sherpa-onnx.node`);
+  } catch (direct) {
+    return firstLine(direct);
+  }
+  return firstLine(error);
+}
+
+/** Downloads `url` into `file`, failing as soon as it exceeds `size` and when it does not match `sha256`. */
+async function download(url, { sha256, size }, file, stallMs) {
+  const hash = createHash("sha256");
+  const controller = new AbortController();
+  let received = 0;
+  let timer;
+  const resetStallTimer = () => {
+    clearTimeout(timer);
+    timer = setTimeout(() => controller.abort(), stallMs);
+  };
+  const tap = new Transform({
+    transform(chunk, _encoding, callback) {
+      resetStallTimer();
+      received += chunk.length;
+      hash.update(chunk);
+      callback(received > size ? new IntegrityError(`received more than the expected ${size} bytes`) : null, chunk);
+    }
+  });
+  resetStallTimer();
+  try {
+    const response = await fetch(url, { signal: controller.signal });
+    if (!response.ok || !response.body) {
+      throw new Error(`HTTP ${response.status}`);
+    }
+    await pipeline(Readable.fromWeb(response.body), tap, createWriteStream(file));
+  } catch (error) {
+    if (error instanceof IntegrityError) {
+      throw error;
+    }
+    const reason = controller.signal.aborted ? `no data for ${stallMs / 1000} s` : describe(error);
+    throw new Error(`download failed after ${received} bytes: ${reason}`);
+  } finally {
+    clearTimeout(timer);
+  }
+  const actual = hash.digest("hex");
+  if (actual !== sha256) {
+    throw new IntegrityError(`sha256 mismatch: expected ${sha256}, got ${actual} (${received} bytes)`);
+  }
+  return received;
+}
+
+async function downloadWithRetry(url, pin, file, stallMs) {
+  for (let attempt = 1; ; attempt++) {
+    try {
+      return await download(url, pin, file, stallMs);
+    } catch (error) {
+      // A wrong digest is deterministic; retrying would only download the same bytes again. A retry overwrites the
+      // file of the failed attempt.
+      if (error instanceof IntegrityError || attempt === ATTEMPTS) {
+        throw new Error(`${url}: ${error.message}`);
+      }
+    }
+  }
+}
+
+/**
+ * Unpacks only the listed archive members, each under its installed name directly in `directory`. A listed member
+ * must be a regular file of exactly its pinned size, checked against its tar header before any of it is written, so the
+ * extracted bytes never exceed the sum of the pinned sizes.
+ */
+async function extractMembers(archive, directory, members) {
+  let mismatch;
+  const unpack = extract({
+    cwd: directory,
+    filter: (path, entry) => {
+      if (!Object.hasOwn(members, path)) {
+        return false;
+      }
+      if (entry.type !== "File" || entry.size !== members[path].size) {
+        // Skipped rather than aborted: an abort fails the pipeline while earlier members are still being written,
+        // which would race the removal of the staging directory.
+        mismatch ??= new Error(`${path} is a ${entry.size}-byte ${entry.type}, not a ${members[path].size}-byte file`);
+        return false;
+      }
+      return true;
+    },
+    onReadEntry: (entry) => {
+      entry.path = members[entry.path].file;
+    },
+    preserveOwner: false,
+    strict: true
+  });
+  await pipeline(createReadStream(archive), bunzip2(), unpack);
+  if (mismatch) {
+    throw mismatch;
+  }
+  for (const [member, { file, size }] of Object.entries(members)) {
+    const info = await stat(join(directory, file)).catch(() => undefined);
+    if (info?.size !== size) {
+      throw new Error(`the archive has no ${size}-byte member ${member}`);
+    }
+  }
+}
+
+async function writeByteTokens(directory) {
+  const source = join(directory, "tokens.txt");
+  await writeFile(join(directory, "tokens.bytes.txt"), spellByteTokens(await readFile(source, "latin1")));
+  // Pointing sherpa-onnx at the original table would silently lose characters again.
+  await rm(source);
+}
+
+/**
+ * Installs the model that `manifest` pins for this host's runtime into `modelDirectory`, replacing what is there only
+ * once everything is in place. Tests call it directly with a short `stallMs`.
+ */
+export async function install(manifest, modelDirectory, { env = process.env, stallMs = STALL_MS } = {}) {
+  const { nativeError, runtime } = await chooseRuntime(env.RIVUS_AUDIO_RUNTIME || "auto");
+  const model = manifest.models[runtime];
+  const baseUrl = (env.RIVUS_AUDIO_MODEL_BASE_URL || manifest.baseUrl).replace(/\/+$/, "");
+  const staging = `${modelDirectory}.partial-${process.pid}`;
+  await rm(staging, { force: true, recursive: true });
+  await mkdir(staging, { recursive: true });
+  try {
+    const { vad } = manifest;
+    let downloaded = await downloadWithRetry(`${baseUrl}/${vad.path}`, vad, join(staging, vad.file), stallMs);
+    // The archive's name must not be one of the installed file names.
+    const archive = join(staging, "model.tar.bz2.download");
+    downloaded += await downloadWithRetry(`${baseUrl}/${model.path}`, model, archive, stallMs);
+    try {
+      await extractMembers(archive, staging, model.members);
+    } catch (error) {
+      throw new Error(`${model.path}: ${error.message}`);
+    }
+    await rm(archive);
+    await writeByteTokens(staging);
+    const selection = { model: model.name, runtime, ...(nativeError ? { nativeError } : {}) };
+    await writeFile(join(staging, "runtime.json"), `${JSON.stringify(selection)}\n`);
+    await rm(modelDirectory, { force: true, recursive: true });
+    await rename(staging, modelDirectory);
+    return { ...selection, downloadedBytes: downloaded };
+  } catch (error) {
+    // A failed extraction can still be closing files in the staging directory.
+    await rm(staging, { force: true, maxRetries: 3, recursive: true });
+    throw error;
+  }
+}
+
+function firstLine(error) {
+  return String(error instanceof Error ? error.message : error).split("\n")[0];
+}
+
+// fetch() reports network failures as "fetch failed" with the reason in `cause`.
+function describe(error) {
+  const cause = error instanceof Error && error.cause instanceof Error ? ` (${error.cause.message})` : "";
+  return `${firstLine(error)}${cause}`;
+}
+
+if (import.meta.main) {
+  try {
+    const [manifestPath, modelDirectory] = process.argv.slice(2);
+    if (!manifestPath || !modelDirectory) {
+      throw new Error("usage: node fetch-model.mjs <manifest.json> <model-dir>");
+    }
+    const manifest = JSON.parse(await readFile(manifestPath, "utf8"));
+    const result = await install(manifest, resolve(modelDirectory));
+    process.stdout.write(`${JSON.stringify(result)}\n`);
+  } catch (error) {
+    process.stderr.write(`fetch-model: ${error instanceof Error ? error.message : String(error)}\n`);
+    process.exitCode = 1;
+  }
+}

package/assets/audio-transcriber/manifest.json

@@ -0,0 +1,64 @@
+{
+  "id": "whisper-1",
+  "smoke": {
+    "input": "model/smoke.wav",
+    "expect": "yellow lamps"
+  },
+  "baseUrl": "https://github.com/k2-fsa/sherpa-onnx/releases/download",
+  "vad": {
+    "path": "asr-models/silero_vad.int8.onnx",
+    "size": 212860,
+    "sha256": "c36d490aff5ab924ca6c7aeec4d8f6bd3d22db6fa17611b9c5b17eae58ac3a20",
+    "file": "silero_vad.int8.onnx"
+  },
+  "models": {
+    "native": {
+      "name": "whisper-turbo-int8",
+      "path": "asr-models/sherpa-onnx-whisper-turbo.tar.bz2",
+      "size": 563790207,
+      "sha256": "b11acbbcd660b44a8e0df33724feb5aaa709cf65668f2823d59f656312544f22",
+      "members": {
+        "sherpa-onnx-whisper-turbo/turbo-encoder.int8.onnx": {
+          "file": "encoder.int8.onnx",
+          "size": 674716297
+        },
+        "sherpa-onnx-whisper-turbo/turbo-decoder.int8.onnx": {
+          "file": "decoder.int8.onnx",
+          "size": 361080764
+        },
+        "sherpa-onnx-whisper-turbo/turbo-tokens.txt": {
+          "file": "tokens.txt",
+          "size": 816730
+        },
+        "sherpa-onnx-whisper-turbo/test_wavs/0.wav": {
+          "file": "smoke.wav",
+          "size": 212044
+        }
+      }
+    },
+    "wasm": {
+      "name": "whisper-small-int8",
+      "path": "asr-models/sherpa-onnx-whisper-small.tar.bz2",
+      "size": 639387718,
+      "sha256": "486a46afbb7ba798507190ffe02fea2dd726049af212e774537efac6afb210a6",
+      "members": {
+        "sherpa-onnx-whisper-small/small-encoder.int8.onnx": {
+          "file": "encoder.int8.onnx",
+          "size": 112442483
+        },
+        "sherpa-onnx-whisper-small/small-decoder.int8.onnx": {
+          "file": "decoder.int8.onnx",
+          "size": 262226114
+        },
+        "sherpa-onnx-whisper-small/small-tokens.txt": {
+          "file": "tokens.txt",
+          "size": 816730
+        },
+        "sherpa-onnx-whisper-small/test_wavs/0.wav": {
+          "file": "smoke.wav",
+          "size": 212044
+        }
+      }
+    }
+  }
+}